What are Head-mounted Displays?
If you’ve ever watched TV or browsed the internet, the phrase ‘Virtual Reality’ probably conjures up a very specific image in your mind: that awesome helmet. Well, they’re real. They’re called head-mounted displays, or HMDs, and they are probably the most instantly recognizable objects associated with virtual reality. As such, they are also referred to sometimes as ‘Virtual Reality headsets’, or ‘VR glasses’. HMDs attach straight to your head and present visuals directly to your eyes, and perhaps most excitingly, to your peripheral vision as well. If a device conforms to these criteria you may consider it an HMD in the broadest sense.
HMDs are not only used in virtual reality gaming, they’ve also been utilized in military, medical and engineering contexts to name a few. As seen in educational films such as The Terminator, these devices can be used to create something called augmented reality, which overlays digital information through an HMD filter onto the real world.
These devices have an incredible range of use. In order to allow for the best possible experience with an HMD, a number of technologies need to be incorporated. Let’s have a look at some of them.
The display is the most obvious part of the HMD, so it stands to reason that we would start here, in the visual center of the device. Today, an HMD might use one of several techniques to transfer images to brain, but the most common is the use of liquid crystals. Sounds painful, right? This is more commonly known as an LCD panel, the same type of panel used in smartphones, modern televisions and computer monitors. A newcomer to the display technology game is OLED (Organic Light-Emitting Diode), which we are beginning to see more and more of already.
Pixels and displays
Pixels (short for pixel elements) are the dots that make up a picture. The more pixels you have in every square inch of display, the crisper the image. According to the late founder of Apple, Steve Jobs, once you have more than 300 pixels per inch (ppi) the human eye can no longer notice the difference, but there are phone displays on the market that are now heading for double that pixel density. For a smartphone, this difference is entirely wasted on the feeble human eye. However, this is great news for VR users, because since the HMD is only inches away from the eye at all times, this high pixel density could spell the difference between fuzziness and (liquid) crystal clarity.
Retinal Projection is a technology that hasn’t seen much widespread use yet, but does exist in some headsets, including the wonderfully named Avegant Glyph. This technology uses a combination of tiny digital projectors and microscopic mirrors to project images onto your retina – hence the name. Many people hail retinal projection over LCD and OLED displays due to its potential for lower eye strain. However, due to current limitations on the technology, retinal projection cannot yet offer the same immersive experience in terms of peripheral vision.
The gamers among us may have heard this term before. The refresh rate refers to how quickly a display can change its content over a particular length of time. Typically, modern LCD computer monitors can do this 60 times per second, or at 60Hz. This also corresponds to a maximum frame rate of 60 frames per second – or 60fps – where one frame equals a complete image on your screen.
As a comparison: cinematic film generally runs at a uniform rate of 24fps, however you may have heard the recent hype around The Hobbit, which was the first film to transition to a frame rate of 48fps. This caused the film to look very smooth or ‘hyper real’, however it has had a mixed reception from audiences – some people report that this newer frame rate is less immersive and feels ‘off’. Over on YouTube, 60fps is gaining traction, especially in ‘action’ footage filmed with a GoPro where motion blur is less than ideal.
To put this all simply; the more frames used in a given second, the smoother and crisper motion appears. Since virtual reality is meant to impose a certain sense of immersion and realism, crisp ‘lifelike’ motion and a lack of motion blur is crucial to the overall experience. It seems so far that 60fps is the minimum needed to achieve this, but HMDs as high as 120Hz are already in existence.
Latency is the term used for the time between input and output. For the layman, that means the time it takes for the picture in your VR world to catch up to your new head position each time. In order for VR technology to fool your brain into thinking that you are in an immersive world, the technology requires very low latencies. An absolutely top-notch experience usually equals latency of 20ms or less; any more and we start to notice an unnatural lag.
If you were to take your smartphone’s LCD screen and hold it right up to your face, it still wouldn’t fill your entire field of vision in the way a VR headset aims to. In order to create that immersive feeling of inhabiting a virtual world, you have to stretch that flat image to fill your visual field entirely. This is commonly referred to as optics. Experimentation by the University of South California indicated that a HMD would achieve the visuals needed for a convincing virtual reality if it had a field of view (FOV) of between 90 and 100 degrees. This effect is achieved by using the lenses in a HMD to take a flat image and change it into something that fills every nook and cranny of our eye line. The quality of the lenses used in your headset is vitally important; an HMD that uses cheap lenses may have a poor picture quality, low clarity, and some unwanted distortion. The old adage that you get what you pay for is true when it comes to the lenses, and one of the most common ways to kick your existing HMD into overdrive is an aftermarket installation of superior lenses.
We touched briefly on head position earlier when discussing latency, and mentioned that the program’s speed in turning your head position information into visual information is vital to the immersion of your virtual world. To acknowledge your head’s position and transform that into other data, your HMD needs some accurate head tracking technology. Thanks to advances in smartphone technology, we can now put a multi-axis accelerometer (which sounds entirely made up, but it is not) on a chip, allowing infrared tracking cameras to accurately watch markers on the HMD and relay positional data to the computer. Mobile HMDs cannot make use of external camera tracking because they are not for use in a fixed location – but some new technologies, such as the Microsoft Hololens, and Google Project Tango, can use multiple sensors in addition to accelerometers for positional calculation.
It is wise to note that some HMDs, especially those that use your smartphone, can only track the direction in which you are looking. Dedicated HMDs with the astonishing capability to track another axis do exist, and they even allow the user to lean in to look more closely at their virtual surroundings. This is a vital extra if you want the full immersion experience.
A logical follow-on from head tracking is eye tracking. For the moment (but watch this space) only one HMD, the FOVE, promises to integrate eye tracking technology, However, third party options are available to upgrade existing HMDs to include such a feature.
Eye tracking, as you may have picked up from the name, tracks your eye movement. It tracks where your eyes are currently looking and can extrapolate that information to change it into visual data. With this technology, your HMD could potentially change the depth of field of the visuals on-screen to simulate natural vision much more closely. Virtual characters could potentially react to your gaze, or you could use your eyes to quickly select items in your virtual world. Eye tracking technology could prove to be crucial for a truly realistic experience, allowing us to interact with a user interface in a much more natural and easy fashion.
Unfortunately however, eye tracking in virtual reality is a technology that is still in diapers, and time will tell the practical uses it will showcase later down the line.
More often than not, you will have the option of using your own headphones in conjunction with your HMD. Some HMDs currently include their own optional headphones, while others do not. Positional, multi-speaker audio that gives you the illusion of a 3-dimensional world is a technology that already very much exists (for example you may already use surround sound somewhere in your home), and it can slot in seamlessly with existing HMDs.
There are three types of HMD when you boil them down. The first operates on a totally self-contained basis. It possesses all the necessary computer hardware required to function as a VR headset, allowing it to collect and display the input it receives from you. These are generally mobile, battery-powered systems that may function using repurposed smartphones – or an actual smartphone – in order to perform its tasks.
The second type of HMD does not have any computing power. This means it must interface with an external computer. Usually these HMDs accept High-Definition Multimedia Interface (HDMI) input and use a Universal Serial Bus (USB) connector to send head tracking data through.
The third class of device is one that does a bit of both, having its own onboard hardware, but also allowing input from external devices.
Although smartphone hardware is, incredibly, at the point where it can provide reasonable virtual reality experiences, they are still a notable distance behind what is possible with powerful computers and major video games consoles. This means that, in terms of pure visual fidelity and frame rate, dedicated external computers are still the best choice for VR. If you think that this means that you must be chained to a desk in order to obtain a worthy virtual reality experience, fear not, that is not entirely true. Wireless display links do exist. However, getting these to work within the ideal latency requirements is still easier said than done.
Now we are left with more mundane things such as the housing and other creature comforts. HMDs are made from all sorts of materials: cardboard, plastic, metal and anything else that will hold the parts together. It’s important to consider what adjustments are available on a particular HMD. The adjustment range of the headstrap is important in this regard. If you wear glasses make sure the HMD will accommodate them or allow for lens adjustments that makes them unnecessary. Finally, the comfort padding and ergonomics of the HMD are often overlooked, but very important. After all, the HMD spends a lot of time strapped to the user’s face.
Companion Input Devices
We’ve touched on head tracking and eye tracking, but for a fully realistic experience we need other forms of input as well, unless you want to stay in one spot without moving around within your virtual world. You can find more info on CIDs here, but for the sake of completeness we’ll run through the basics below.
At present, the most common way of navigating virtual worlds is with existing videogame peripheral devices such as gamepads, flight sticks, racing wheels, or the old classic; the keyboard and mouse. More immersive devices meant specifically for VR are on their way, however, such as omnidirectional treadmills and specialized devices like the SteamVR controller.
At the very high-end you will find full-body suspension and motion tracking systems, hydraulic vehicle simulation rigs, and active mechanical force feedback tech (or ‘haptic technology’), which ‘reacts’ kinaesthetically to your virtual environment, like the rumble packs on your console controllers. This technology is fantastic for creating immersion when it comes to walls, barriers, and other virtual environmental features. Without these immersion additions you might otherwise feel as though you’re slipping right through solid objects.
So How Does It All Work?
Momentarily setting aside less common technologies like retinal projection, most HMDs that use LCD or OLED displays work by presenting each eye with an almost identical, but slightly offset, version of the same image. This provides the illusion of stereoscopy – surround sound for your eyes! Or, you know, ‘3D’. Most HMDs save on dual-screen costs by utilizing a single screen with a plastic divider, to ensure that each eye can only see its specialized image.
If you were to remove yourself from the device, you would see that the images do not fill the display from edge to edge, and that they have fuzzy grey edges. This is a simulation of our actual visual field, with the sharp image at the center and curvature and gradual loss of acuity towards the edges. Viewed through the special lenses and at the right distance, these pictures neatly fit into our visual field and appear natural; a perfect simulation of looking at a real scene, not a picture.
When all of this comes together you will no doubt feel as though you are truly present in a virtual world. Wherever you look in your HMD, you will see a virtual reality seamlessly replacing the real world around you.
This was a broad overview of HMDs. Be sure to check out our in-depth articles on individual HMD products that are on the market currently, or in development. Now that you are armed with all the necessary knowledge, you should have no trouble understanding the range and variety of the devices on offer.