Motion tracking, the process of digitising your movements for use in computer software, is incredibly important for virtual reality. Without it your virtual self is paralysed, unable to move its head or move around. At best you’d have to use an abstract control system like a gamepad instead which, while functional, does your sense of immersion and presence no favours.
Putting our bodies into virtual worlds is therefore essential to a first-class, immersive experience. Of course, there are degrees of immersion at work here. If you are simulating sitting in the cockpit of a helicopter, space ship or other vehicle there is no need for full body motion capture. If you want to get up and explore your spaceship however, the spell will be quickly broken.
There are many technologies and methods available when you want to get yourself into the virtual world, but what are they and where do we begin? Let’s have a look at the basics of motion tracking.
What Does Motion Tracking Actually Track?
Before we get into the actual hardware, have a look at the picture above. It describes every possible way an object can move in 3D space. You’ll notice that there are six types of motion each with its own coloured arrow.
These are referred to as the six-degrees-of-freedom (6DOF) and any tracking systems that intends to provide a complete motion tracking experience needs to measure movement along all of these degrees. You’ll hear the term 6DOF quite a lot in reference to virtual reality hardware.
What Technology Underpins Motion Tracking?
There are quite a few ways motion can be tracked, here we’ll break them down into two broad categories: optical and non-optical tracking. Optical tracking is where an imaging device is used to track body motion. Non-optical tracking uses a variety of sensors that are often attached to the body for the purpose of motion tracking, but can also involve magnetic fields or sound waves.
Many motion tracking systems use a combination of these methods for added accuracy, but as the sensors improve we are seeing systems that are more compact and accurate.
Optical Methods of Motion Tracking in VR
Optical methods of motion tracking usually use cameras of one sort or another. The person being tracked has optical markers. Usually dots of highly reflective material on certain known points of their body or on the equipment such as the HMD or handheld controllers. In professional contexts where motion is captured for use in animation an actor’s body may be covered in such markers, but commercial systems designed for personal use in virtual reality may use only a few strategic markers or even no markers at all. When a camera installation capable of calculating depth sees a marker it can map it to 3D space. For example, two cameras at known angles that both see the same dot allow for this mathematical calculation. One issue that does arrive with this method is that of maker swap where two cameras see different dots, but think they are the same one. Obviously this provides incorrect tracking data and inaccurate motion tracking.
Reflective markers are known as passive markers as they reflect light, but there is another type of marker known as an active marker. These are computer controlled LEDs that allow for more accuracy and various workarounds for the weaknesses of passive marker methods. LEDs may be different colours or flash rapidly in sync with the capture system Although active marker systems perform better than passive ones, for the subject there are drawbacks. The user has to wear a power supply or be tethered to the system somehow, which is clearly an encumbrance.
Active optical tracking is not prevalent in consumer virtual reality at present. HMDs with markers, for example, have passive ones. One notable example of active tracking in a consumer device is the Playstation Move controller, which has a large colour changing light on one end. This is tracked by a camera connected to the playstation console. This tracking isn’t exactly the same as the active systems described above, since there is no depth tracking. The move controllers do however contain several non-optical tracking systems that augment the camera based tracking.
Arguably the best known and most successful consumer motion tracking device is the Microsoft Kinect, this sensor package designed for the Xbox 360 and One consoles as well as Windows computers is capable of performing complex motion tracking without the use of any markers at all. The camera simply needs to see the subject. The latest version of the Kinect can track up to 6 individuals simultaneously. These capabilities are achieved through some very clever software methods combined with special infrared and depth sensing cameras, but still cannot attain the precision of professional marker-based systems as of yet.
The Leap Motion is another example of a marker-free tracking system, but rather than full body tracking the Leap Motion creates high-resolution real time scans of objects in close proximity to it. In virtual reality contexts the device can be attached to the front of an HMD and provide precisely digitised versions of the user’s hands, which then allows for natural interaction.
Full body tracking that rivals professional systems in terms of movement freedom and accuracy for the consumer are not yet with us. At the moment most virtual reality for consumers is a “sit-down experience”. Products such as the Oculus Rift are designed for this sort of experience at the moment. Oculus itself recommends that you don’t stand up and wander around while wearing their HMD, since you’ll likely trip over something and hurt yourself.
However, the upcoming Lighthouse tracking system, which is part of the SteamVR ecosystem, may begin to change this situation. The Lighthouse system is a laser-based tracker that can see passive markers on SteamVR HMDs and controllers. It can map objects in the room and provides safety measure to prevent injury while moving around in virtual reality.The Lighthouse system promises to be revolutionary, but time will tell how well it works in practice.
Non-optical methods of motion tracking in VR
The virtual reality equipment that you wear on your body such as an HMD or the controllers you hold such as the Playstation Move, Oculus Touch or SteamVR controller contains micro-electromechanical sensors such as accelerometers, gyroscopes and magnetometers.
For a more complete discussion of these three sensors be sure to have a look at our article on Understanding Sensors, here it is enough to say that these are microscopic devices that can measure the lateral, rotational and compass orientations of whatever they are attached to. The gyroscopes measures 360 degree rotation, accelerometers measure movement along the XYZ axes and magnetometers can determine orientation towards a magnetic field. Which means they can tell which way magnetic North is for example. Thanks to developments in the automobile, aeronautical and computer industries these devices have become cheap, accurate and positively tiny compared to the full sized devices that came before them. It’s especially the success and growth of the mobile device market that has driven the modern development of these sensors and it isn’t an exaggeration to say that the new generation of consumer virtual reality systems owe their existence to technologies developed for smartphone and tablets.
Combining these three sensors can provide a device with low-latency, precision motion data. This can be combined with optical methods such as infrared tracking or passive reflector tracking for truly robust motion tracking, but for untethered, self-contained systems these sensors can do an adequate job by themselves. As evidences by various mobile virtual reality products.
Other non-optical tracking technologies are somewhat more exotic. Direct electro-mechanical sensing of body movement is one such technology. Some modern version of the virtual reality glove and other haptic products also double as direct electro-mechanical motion trackers (see our article on haptics). When you flex your fingers sensors in the glove are activated and convert that movement into electrical signals for motion tracking. Examples include the GloveOne and the Salto.
Another interesting approach, which is not virtual reality specific, but has potential in this area is the Myo armband. The Myo interprets electrical impulses from the muscles in the forearm allowing for gesture-based controls of computer software. What makes the device special is how unobtrusive it is, since it is designed to be worn all day. Perhaps in the future such direct sensing could be precise enough to replace other tracking methods and it is a technology worth keeping an eye on.
Motion can also be tracked using systems that are more overtly mechanical. Mechanical exoskeleton systems can be used both to provide haptic feedback and to capture motion. For example, the Gypsy motion capture suit includes worn components that directly measure movement with potentiometers. The Dexmo F2 is a hand worn exoskeleton that can provide both force feedback, positional tracking and hand motion capture. Another type of mechanical movement tracker is the omni-directional treadmill. These have been used in military virtual reality systems to allow soldiers the freedom to run infinitely in any direction, but consumer versions are in the making. The most well known is probably the Virtuix Omni, a treadmill meant for home use that can measure both running, crouching and jumping in a limited fashion.
The Future of Motion Tracking & VR
These optical and non-optical motion tracking methods are likely to be in motion tracking systems for the foreseeable future. However, what if you didn’t need your body to move around in virtual reality at all?
Even today the development of direct brain-machine interfaces is fairly advanced. In the area of neuroprosthetics, for example, it’s already possible for quadriplegic individuals to operate robotic arms to accomplish complex tasks. Using something called targeted muscle reinnervation, the truncated nerve endings of amputees can be rerouted so that robotic prosthetic limbs can read signals from them and move the way natural limbs do.
At present these control methods require invasive surgery, but advances in sensor technology and further miniaturisation of electronics promises a future where we don’t move a muscle in the real world, but can freely move our virtual bodies using nothing but our minds. It may seem far fetched now, but the basic premise has been demonstrated, so the idea is at least plausible.
Today we must still make do with limited motion capture and good old peripherals to let the virtual world respond to us, but there are certainly exciting times ahead.