Augmented Reality – What is it?

Although this site is dedicated to virtual reality, you cannot discuss it without mentioning its very close cousin augmented reality, but what is it?

Whereas virtual reality immerses your senses completely in a world that only exists in the digital realm, augmented reality takes the real world of the present and projects digital imagery and sound into it. Augmented and Virtual Reality both fall on the continuum of mediated reality. Which is where a computer system modifies our perception of reality versus the “real” world.

As you can probably deduce this means many things qualify as augmented reality. The heads up displays we see in some aircraft and cars that may show you things like “distance to a target”, GPS position or your current speed are a form of augmented reality. Events with digital avatars of deceased musicians such as Michael Jackson and Tupac Shakur projected onto a screen using the Pepper’s Ghost illusion would also qualify under a broad definition of augmented reality.

However, when we hear about augmented reality these days it usually refers to a much more sophisticated, interactive and spatially aware implementation of the concept. Where digital objects such as 3D models or video are projected onto our view of reality as if they were really there.

How Does Augmented Reality Work?

The type of augmented reality you are most likely to encounter uses a range of sensors (including a camera), computer components and a display device to create the illusion of virtual objects in the real world.

Thanks to the popularity of smartphones, which have all the necessary components, they have been the place most commercial augmented reality applications that have been released.

In general the device looks for a particular target. This can be anything, but usually it’s a 2D image printed on something like a movie poster. Once the augmented reality application recognizes the target via the camera it processes the image and augments it in some way with pictures and sound. For example, you may see the movie poster spring to life and play a trailer for the film. As long as you look at the poster through the “window” of the display you can see augmented reality instead of plain old vanilla reality.

By using smart algorithms and other sensors such as accelerometers and gyroscopes the device can keep the augmented elements aligned with the image of the real world.

Using a smartphone or tablet computer as a “magic window” into the augmented world is one way we can relay this digital info to our eyes, but there are many other ways to achieve this.

Digital imagery can be projected directly onto physical objects. This is known as projection mapping and can be used to quite striking effect. For example, the Dyadic Mano-a-Mano uses projectors and Microsoft Kinect sensors to provide the user with 3D digital imagery projected directly onto the environment. The user doesn’t need to wear equipment or use any devices. Interaction with this system is highly natural and intuitive.

Projection mapping for augmented reality

Projection mapping as an augmented reality method has a lot of potential, but it requires a controlled and mapped space in order to work. The method that is most likely to supplant smartphone augmented reality as a common implementation outside of the laboratory is one that uses head mounted systems. This is where virtual and augmented reality really begin to converge, as there is no real reasons why the head mounted systems used by both technologies cannot be cross-functional. Indeed, head mounted systems that use smartphones to work often have something known as a camera “pass-through”. In other words, although you can’t see anything other than the screen of the head mounted display (HMD) it can show you the outside world via the rear-facing camera of the phone. This of course allows for augmented reality without the need for a handheld device. However, unless specially designed against it, this method leaves one feeling a bit disconnected from the experience, since the camera’s perspective and lack of depth perception don’t quite gel compared to what the naked eye sees.

One way to get around this is by using a system as found in the Google Glass and Microsoft Hololens. Both of these devices use something known as a “prism projector”. The eyes of the user look out at the world unimpeded, but digital imagery is projected into the prism projection system that sits between the eye and the outside world, making it appear as if those objects are really there, sitting on a table or hanging against the wall. The way this HMD achieves this is complex and fascinating and it discussed more completely in our Hololens article.

There are many ways to achieve the goal of augmented reality, but as you can see the end result is that we see digital information blend with the analogue world. Something that has many, many applications. Some of which we’ll take a closer look at.

Applications of Augmented Reality

Augmented reality has a wide range of applications in several industries and thanks to the rise of consumer smart devices and overall computing technology developments it now has lots of potential in the mainstream consumer space as well.

The two areas where we have seen a lot of commercial development in augmented reality are education and gaming.

The two biggest mainstream video game consoles, the Xbox and Playstation, have included augmented reality capabilities for the last two console generations. These game in the form of the Kinect (for the Xbox) and Playstation Eye or Camera (for the Playstation 3 and 4 respectively). Because you’re facing both the camera and the screen in these implementations are more like augmented reality mirrors, where you see yourself “in” the game and can interact with game characters that look to be in the same room as you.

Mobile augmented reality games are also not rare, and can be found on smartphones, tablets and handheld consoles such as the Nintendo 3DS and Playstation Vita.

Seeing the potential for augmented reality in education isn’t hard. It’s being implemented in fields such as medicine where students can benefit from live 3D models. It’s possible to use existing learning material (such as pages from a textbook) as targets for augmented reality. So when viewed through the lens of a smartphone you can see that picture of an engine animate in an engineering textbook or a working 3D model of a beating heart that you can walk around of rotate by hand.

In medical practice augmented reality can project information directly onto the body of a patient. For example, the Veinviewer system projects a real-time image of infrared vein scans directly onto the patient’s skin. Creating the impression that the skin is transparent. This allows the clinician to “see” the veins directly.

Military use cases are also quite clear, since soldiers wearing heads up displays (HUDs) can see information tagged onto objects in the real world. Radar information, orders or any other relevant sensor data from devices on the network that can provide it. Enemy and friendly positions are of course also useful to know. Augmented reality clearly has a bright future in military applications.

Mobile phones especially the iPhone use augmented reality apps which allow you to view computer generated images that have been superimposed over real world images. An example of this is an app which helps you to find a restaurant: it does this by displaying restaurant signs/logos as you move in a particular direction.

Another useful type of app is a golf GPRS system which helps golfers around a course. It displays yardages for each of the 18 holes, shows where the hazards are, e.g. bunkers and advice and support on improving your game. If you are golfer then this app will appeal to you immensely – look for the Golfscape Augmented Reality Rangefinder from the Apple store.

Augmented reality is also used in marketing and advertising as a means of enhancing certain aspects of a product in order to make it more attractive which will boost sales. This is discussed in more detail in our augmented reality marketing article.

Conclusion

Augmented reality is likely to worm its way into our daily lives more and more in the 21st century. Once wearable computers become more common it won’t be strange to see people interacting with and reacting to things that aren’t there from your perspective. Thanks to technologies such as augmented reality the way we work with computing devices and think about the divide between digital and analogue reality is likely to change fundamentally. Nothing is stopping you from experiencing augmented reality for yourself today though. Just hop onto your smartphone’s app store and search for “AR” apps. There are plenty to try, many of them free.

A few of these topics such as augmented reality games contain a series of smaller sub-articles, for example augmented reality games for the iPhone.