Prototyping - Nintendo Virtual Boy 2

This is an archived repost of an article i wrote for The Nintendo Enthusiast in June 2014, but the vast majority of the theory is still surprisingly relevant to VR design:

Ever since the Oculus Rift and the 3DS came along, I’ve been wondering what Nintendo would do if they tried their hand at the Virtual Reality game (again). They pioneered motion control, made wireless video streaming lagless, and pushed Augmented Reality further than anyone else. However, their drive for inclusive gameplay is at odds with the very insular world of modern VR systems. But what if there was a way to make VR more social? Could the Virtual Boy rise again, and redefine VR in an iconically Nintendo way?

(I realise now that this requires a lensing layer, but the article does pre-date Google Cardboard!)

The current crop of VR headsets provide great new possibilities for diversifying gameplay, but they’re still hindered in many ways: They’re wired, expensive, tend to be peripherals for a larger system, require further peripherals to control properly in the VR space, and are also slightly unsafe to use as they occlude your vision and sense of orientation (case in point).

A few months ago I set out to develop a hardware concept which addresses these issues using nothing but existing technology, and all whilst being backwards compatible as a 3DS/DS. The novel design not only stabilises the 3D effect of the 3DS (a personal bugbear of mine), but allows for brand new ways of experiencing virtual and augmented realities. It even touches on Nintendo’s intentions to establish a Quality of Life platform.

I can\’t stress enough that everything I’ve used in this device exists right now! The screens are from the 3DS XL, as are the cameras, game cart slot, gyro/accelerometer/geomagnetic sensor, and controls. The sensor bar tracking technology comes from the Wii. The battery and bluetooth connection is straight out of the Wii U Pro Controller. The case is deliberately designed to be ”too big” to make generous allowance for the internals, much like the original design of the DS. I have every belief that this thing could be less than half the thickness I’ve used, but that would make the gamepad less comfortable, and i’m not in the business of making wishy-washy concepts which are wafer thin and run on magic.

At 185x78x35mm it has a very similar footprint to that of the PSP, albeit somewhat thicker. These dimensions are limited by the comfortable width of a headset (in much the same way that sunglasses are), and the functional focal distance of the human eye.

So here’s my concept for an improved 3DS XL which readily converts into a VR/AR headset, complete with a wireless gamepad! Dual Screen gaming was great, but it’s time they parted ways. Here's the Virtual Boy 2!

Looks like a 3DS, but with an extra control stick! Oh if only the real 3DS had one...

No added parts to plug in, it's always with you and ready to go!

Animating the head strap was going to be a chore within Sketch-up, but it just stretches out of the way.

Let me break it down for you (then fold it up).

Lower Half:

Essentially the Wii U gamepad, but without the wireless video transmission. When disconnected from the top half of the device the screen goes blank. This drastically cuts down the power usage, allowing it to run a much lighter battery. Like the gamepad it features a full suite of motion sensing technology (gyroscope, triple-axis accelerometer, and high-fidelity geomagnetic sensor), but it also houses an IR sensor bar which allows the headset to track its absolute position in 3D space. This means that the bottom screen can be seen as ”on” through the headset, like a much more accurate version of the AR cards which came with the 3DS.

In terms of physical control, the device packs two analogue sliders for symmetrical control options with legacy support of the circle-pad pro built in, the venerable d-pad, a/b/x/y buttons, and two shoulder buttons. The resistive touch screen also still works in spite of the display not operating.

The wireless connection between the bottom and top half is Nintendo’s bluetooth standard. This means it’s also possible to use Wiimote controls for some headset-based games (VR Wii Sports, anyone?), with the gamepad even doubling as an impromptu sensor bar for body position tracking.

Because the bottom half of the device houses none of the significant hardware, the stylus slot can run the entire width of the device, allowing for the stylus to be loaded from either side. Much like the symmetry in the controls, this is a boon to lefties everywhere, and makes the device more comfortable and customisable for more of the population. There is also ample room for healthy, strong rumble motors in either side of the device, and a several-hour LiPo battery.

Rounding out the features are the obligatory charger and IR ports. I decided to leave the charge port off of the headset to ensure no wire-based injuries arose, and to make sure the gamepad battery always outlasted the headset (or you’d need to connect both halves to separate chargers to play on AC power!).

The over-engineered hinge mechanism lets the two halves split, much like like the Asus Transformer line of tablet devices. Here, you press the DS symbol button to unlatch the top half, a symbol which denotes two screens. Fitting.

All up weight for the bottom half could be as low as 80g, although I’d recommend increasing the heft to give it better hand feel. Fitting in as much battery as possible would allow the bottom half to charge the top when connected (always leaving a 2 hour reserve), again as seen on the Transformer line of tablet devices.

Upper Half:

To reduce latency and overall power usage, housing the major hardware along with the primary display makes the most sense. As such, the top half of this device includes the typical ”guts” of a 3DS; the gpu/cpu, game cart slot, and SD slot (micro in this case). The rear of the device houses the typical 3DS stereo cameras, although bumped in resolution and capture rate. In IR mode they function as the Wiimote camera, tracking the gamepad’s sensor bar with the Wii’s famous speed and accuracy.

In addition, the top half houses its own gyro/accelerometer/geomagnetic sensor unit. This allows for the traditional Oculus Rift head movement tracking, and when used in combination with the gamepad gives full separation of movement between your hands and your head.

Because of the processing power required, this half will house a significantly larger LiPo battery, which will be split in two and housed in the speaker pods on either side of the device. This means that when in headset mode the CoG of the device is as close to the face as possible to further improve comfort.

The power and notification lights wrap around the front face of the device to be viewable whether the device is closed or not. The Notification light (messages/low battery warning) also protrudes slightly into view when in headset mode, creating a small glow in the corner of your vision when required.

The obligatory microphone is housed on the bottom edge of the device. This ensures that when in headset mode it is isolated from the speaker output, boosting clarity.The speakers on the device are fabric-covered and semi-parabolic.

When in 3DS mode they project sound forwards, yet when headset mode they tunnel sound along your cheekbones and into your ears. The curved gap they produce along the edges also provides clearance for glasses, should they be required.

The left edge provides volume controls and a 3.5mm headset jack, keeping the audio settings together. The right edge houses the 3D-level control for setting viewing depth. All controls are clear of the headband while in headset mode, allowing easy access while in-game. While i do prefer physical sliders for these settings, the headband may bump these, so digital buttons it is!

The top edge houses the game-cart slot. I was keen to make this a ”download only” device, but we all saw how that worked out for the PSP Go. I have a feeling that the Go, like the original Virtual Boy, was a victim of being ahead of its time. Dropping the cart slot would free up an unbelievable amount of volume from the device, but Nintendo loves their backwards compatibility, and so do I!

Looking at the back of the device, an elastic headband runs flush, terminating at ball-swivels on either end. Flipping this headband over to the front of the device causes the speakers to pivot inwards, initiating the TRANSFORMATION!

Once the headband is out of the way, the top plate can be rotated 180 degrees, slid along its spring-loaded track, and clicked into its final position.Then the bottom half is rotated in a similar fashion, and likewise clicks into place. The adjustable silicone nose strap flips up, and you’re ready to go!

The edges on both plates are trimmed with a comfortable rubber foam, and the points of contact are above your eyebrows (on top of the brow ridge), and just above the fold of your nostrils. This shape will accommodate the vast majority of human faces, from young teens up. The product is obviously NOT suitable for small children!

Once converted, the ”face” on the back of the device is defined by the stereo cameras. I can see the potential for a million limited edition devices coming from drawing elements onto this face (Dragon Quest’s slime comes to mind!). It also means that when worn, the user still presents a face of sorts to the surrounding public. Maybe it shouldn't be quite so grumpy, though!

All up, this half of the device should come in at under 200g (an entire 3DS weighs 235g, and the XL is 336g). This estimate is comfortably high however, especially when you consider the weight to power to battery life of modern mobile phones. My all-metal HTC One has a similar screen size, weighs 160g, and packs a 2600mAh battery (twice the capacity of a 3DS!). A fully optimised, single PCB system could probably get under 150g while providing 4 hours of VR-mode gameplay (which is PLENTY of time to neglect the real world).

Now, onto the most important part, the software (because it’s how you use it that matters!).

Depth Processing:

Like humans and my HTC phone, the 3DS has stereo cameras. This means that it can use depth perception to know how far away things are. My camera uses this information to blur or colour objects which aren’t in the ”focused” part of the depth spectrum, creating the ”bokeh” effect all of the smartphone manufacturers crave. This can use that information for more intersting things, things like ”blue screening”:

Here’s a photo of my arm. It’s in a room. The stereo cameras can see that my arm is closer than the wall because the arm is shifting significantly more between the two images. This is like holding your finger in front of your face and comparing the view of your left and right eyes. Given this information, software can relatively easily cut out objects which are obviously far apart, and then overlay them into another world, in real time.

This is neato. Finally, the clumsy animations in an Elder Scrolls game can be your own!

Virtualised displays:

The 3DS was the first device which properly sold me on Augmented Reality. Until then, phones could use it to show you were an ATM was, but it was clunky and the flat display fooled nobody. The 3DS used stereo cameras and a 3D display to make it appear as if you were looking ”through” the device. Coupled with the AR cards which fairly accurately defined a plane in virtual space, the illusion created was quite gratifying. But Nintendo can do more…

The Wiimote used an IR camera which occluded all other light waves to simply look for two dots. This was a far more efficient tracking method than was used for the PS Move or other ”full vision” tracking methods. By integrating a low-power sensor bar on to the gamepad half of the device, the stereo cameras can easily keep track of the positioning of the gamepad, with the gamepad sensors providing additional orientation data and ”filling holes” when the vision is temporarily obscured.

Now we can essentially overlay any AR data we want on to a moving surface, with great accuracy. This means it’s possible to ”project” a display on to the gamepad’s screen, making the intensive Wi-Fi streaming of the Wii U’s gamepad entirely unnecessary (ok, so no local multiplayer, whatever, it’s a handheld!).

We can even go further, and project anything we want on to the AR ”bed”. You could have a 3D Pokemon in your hands, doing battle against a friend’s across the room. It’s basically the Pokemon TV ads, come to life!

This is also neato.

Safety:

I know safety should always be first, but here it’s third, so what?! Headsets like the Oculus Rift are immersive, REALLY immersive. The trouble here is that while wearing a headset, your concept of up and down is almost entirely overridden by what you see in the game (in much the same way that pilots use their virtual horizon). This inability for your body to resolve what is right can get VERY messy when your brain is fed conflicting information:

Because we have stereo cameras on this headset, we can ”save” people from their virtual world by switching the visual feed to that of the real world. If the motion sensors detect an ”accident”, they can override the game. Obviously this threshold will need careful adjustment in terms of how it’s employed, but it’s a necessary consideration. The cameras can also function to warn users of impending collision. Using their depth-sensing ability, the cameras can detect close objects. With more advanced hardware they should also be able to detect the presence of recently moving objects, sparing you from stepping on the cat while immersed in Ubisoft’s latest VR dancing game!

This isn’t so neato, but it’s important to the widespread appeal of the device.

Mii Integration:

Mii’s have become pretty important to Nintendo. They can get fat in Wii Fit, they can be different weight classes in Mario Kart, and they can even take on a myriad of skills in the new Smash Bros. game. In the world of VR however, we can now assign a real height to Miis, and have that reflected in-game.

Not only can you experience virtual worlds at the height to which you are accustomed in reality, but you can also, for the first time ever, understand what it’s like to be a person of different height! You could finally see what the world looks like from Shaquille O’Neal’s perspective! I’m sure there are meaningful, deep, and moving applications for this feature, but I’ll leave that to the indie game designers who specialise in ”feels”.

Wirelessness(ess):

Having no wires lets you do really cool things! It lets you use VR or AR wherever you want. It lets you play a virtual FPS game in an actual space. It lets you add special effects to your LARPing (PvP in P?). It lets you Watch for Yokio’s in the real world, and makes Digimon pretty much a reality. It also lets you watch VR porn on the train to work, which explains the face this system has… but let’s just not go there, Nintendo is family friendly DAMMIT!

With these tools, game developers could come up with all sorts of exciting and new gameplay innovations. Here are a few of the applications which I’ve had bouncing around my noggin:

LARPing:

This could make mainstream and cool that which traditionally involves adults running around a field with tinfoil-covered sticks and throwing small bean bags at each other (Live Action Role Playing). You see, what I propose is that we put the AR into LARP (ah, now you see, eh?).

When wearing the headset, players can utilise the ad-hoc networking capability of the device for local multiplayer. PvP could easily be started by having both combatants bow, with the motion sensors translating that movement into the ”handshake” for Wi-Fi pairing (as used in mobile devices for less interesting tasks).

Battles could then be fought in a turn-based style ala Pokemon, allowing the digital tracking of stats, levels, and gear to translate into the real world. They could also use a blend of turn based and realtime combat, as found in the Super Mario RPG series, or draw spells and counter spells using either the touch screen or motion control of the gamepad.

Armed with a GPS receiver, LARPing battles could even be viewed, tracked, and commanded from a top-down perspective on another device, adding an RTS elements and asymmetrical multiplayer into the mix! World of Warcraft meets Warcraft 3 meets the Renaissance fair, anyone?

Or we could be a little less ”hardcore”, stick with AR Pokemon games, and start avoiding long grass in real life (which reminds me of the surprisingly accurate Pokemon pedometer, the Pokewalker).

F-Zero:

All VR systems need a good vomit-inducing thrill ride! Strap the headset on, hold the gamepad out in front of you, and try to hang on to your lunch as you play F-Zero from the first person perspective! You\’ll finally be able to look in your mirrors or over your shoulder at enemies, or reading the track ahead as it twists and turns at stupid speeds.

Using the Stereo cameras, we can even put your arms and shoulders in the game. An occlusion distance can be set, putting your torso in the cockpit in realtime, and critical information can either be on your HUD or projected onto the virtual gamepad display.

If Nintendo hasn’t released a new F-Zero because they haven’t got any new ideas for the series, this should do it!

Fatal Frame:

Ok, the Wii U is getting a new Fatal Frame at some point, and that\’s cool because the Wii U is an impressive VR camera. But if you really want to test your trouserly fortitude, you can’t go past VR.

In this concept, you’re free to look in any direction in a fully-realised VR world. Movement is handled by the gamepad in relation to its orientation. What that means is that if the gamepad is pointing North and you're pressing left, you’ll be strafing West regardless of which direction you’re looking. The separation of looking from movement also means you can walk North but look South, which is handy because you’ll be looking over your shoulder quite a bit!

When you hold the gamepad up to your eye level, the stereo cameras pick up the sensor bar on the gamepad. This puts a virtual, actual camera in your hands in the game. You can look at the live LCD screen on the camera and snap the ghosts which torment you.

The flash effect could be brilliantly disorienting in this one, whiting your vision but leaving a burnt after-image of the ghost in your view. Ooh i want to play this, but in a super safe environment, and over a plastic sheet.

Traditional FPS:

The obvious choice. You can look in any direction, and the gamepad works like the Wii U Zapper allowing you to shoot in any direction. Hold the gamepad up and you can look down the sights. Very accurate, very fast, but probably more physically demanding than your average CoD player is willing to commit to. I don’t care, it should be epic fun!

You could even pair this with a gun peripheral which the gamepad slots into. With the gamepad in line with the gun, you could move with your forward thumb on the analogue slider, and have a d-pad and trigger on the peripheral for your back hand. I love peripherals!

Football Field FPS:

This requires the inclusion of a GPS receiver in the device, but it lets you play paintball-esque online shooters. All you need is a big, open space with even ground. Instead of using the analogue slider to move, you use human power!

The game starts by telling you to walk to a specific point on the field, and point in a specific direction (for example: ”stand in the centre, face a goal post”). From here, the system knows exactly where you are located geographically at all times. You are then presented with an entirely VR world in the headset, like a virtual mouse maze laid over the bare field, and then you\’re let loose to frag your opponents (who may be playing on the same field as you, or playing remotely!). The location of all players is tracked and synced, and you’re basically able to play laser tag, with cooler graphics, without the requirement of actual friends!

The stereo cameras will detect and alert the player of any looming danger (a goal post, person, etc).

Of course, because the walls are virtual, people who don’t respect ”clipping” will have to be punished in some way. I think going with the Tron-like ”walls are insta-death” is fair enough. Also, maps can only be flat levels for obvious reasons.

If you were super-committed to this, it would be possible to make a real-world map layout which is then copied in the virtual, allowing you to lean on walls, go up stairs, etc. Laser tag businesses may want to look into that!

My end game for this gameplay would be a laser tag arena which was essentially Left4Dead, where a group of real people takes on a virtual zombie horde. That could look amazing! A much cooler workout than playing Zombies, Run! on my phone.

I shall coin this the FFFPS, and the Marathon perk unlock will truly be earned!

While i do think I’ve come up with an interesting product, I do realise that this is an imperfect solution. While this device can provide an immersive 3D experience, it doesn’t quite offer the enveloping view of an Oculus Rift. Instead it’s like looking through the visor of a helmet; it’s still able to confound your senses, but it puts a pane of glass between you and the world.

As with the Oculus Rift, improving the resolution of the screen would be a high priority. Copying Apple’s ”Retina” technology and doubling the resolution would ensure the backwards compatibility of the device while allowing for the more impressive close-up viewing required of a headset.

Finally, with technology improving as it currently is, the ability for Virtual Reality and Augmented Reality to seamlessly enter our lives is approaching quickly. Already, companies like Google are introducing AR Glasses technology (Glass) and portable 3D world scanning (Project Tango). Social, inclusive VR is coming one way or another, and it will change our lives far more than the Oculus Rift ever will. While it took a critical step forwards for the fledgling field, it is not even close to the end point of the journey. The narrow scope of its application will make way for more ”Nintendo” oriented devices, and gaming will be an integral part of that change. I never thought I’d say this, but I’m keen for a Virtual Boy!