Oculus is Opening Up Experimental Mixed Reality Access on Quest
Both Oculus Quest devices are festooned with cameras that allow the headsets to track the world around you for accurate positioning. If you’re a Quest user, you know that you can actually see the world as those cameras see them by switching to “passthrough” mode. This is a key safety feature so you can quickly get environmental awareness without taking off the headset. Until now, this has been a part of Quest functionality limited to system access only. In other words, third-party developers couldn’t write apps that are meant to take advantage of passthrough mode.
Now the company is releasing a new API that will allow developers to make experiences set in the mixed reality space. That’s the same mode where you see your guardian overlaid in the real world. Before, that part of the system was off-limits. Giving developers access to what’s going on behind the curtains is a big deal for everyone, including us as users.
What the API Does
Essentially, this new API lets developers use the data from Quest’s external cameras within their application. How it will be used obviously depends on the creativity that each developer applies. However, the buzz from Facebook and Oculus seems to lean towards productivity applications.
What’s really interesting is that developers can selectively blend in some elements of the real world. So it can be used for applications that are mainly VR, but with some elements of mixed reality.
We’ve already seen some inkling of this with Quest experimental features that let you map your real couch or keyboard into the VR image. So you can imagine that there are many different VR apps that could benefit from selective mixing of VR and the real world.
A Black and White Issue
One fly in the ointment is that the cameras on the Quest only offer distorted black and white images. After all, they were designed for efficient motion tracking, not as mixed reality sensors.
While I think you could untwist the image using a bit of clever mathematics, the only solution for the black and white image would be some sort of false colouration. The real solution, in the end, will be to only bring small bits of the real world into the VR space. Perhaps the next iteration of Quest (or the Quest Pro) will feature colour cameras to allow for a more complete mixed reality experience.
Privacy Issues
Giving a developer access to a camera feed of your home obviously raises issues of privacy. Oculus has addressed this by applying the same rules around private data that already protects your privacy on Quest.
All processing and storage of data from the cameras never leave your headset. Developers can’t request it or ever see what your camera sees.
Who’s This For?
Given the weird black and white world, you see in passthrough mode, I don’t think anyone who’s excited for something more like a Hololens should get their hopes up. The main use case I see for this is letting you use a real keyboard and mouse while working in an app like Virtual Desktop.
As for games and other experiences, it’s less clear how you could use the feature. Perhaps other real-world objects could be integrated into games and there’s always the possibility of software-based processing of the black and white image. Whatever developers do with this new level of access, we’re all eager to see it.
