How Does that Make You Feel?: Facial Tracking Comes to VR

There’s an exciting branch of computer technology known as affective computing. Yes. that’s “affective” and not “effective”.

Affective computing is all about how we can integrate human emotional states into computing. It includes both how computers can read and understand our emotions as well as how they can reproduce them in different ways.

Why would anyone want this? Well, think about what your technology could do for you if it knew what your emotional state was at a given time.

In video games it could really change the immersiveness as characters react to how you feel about them. For more serious applications it could mean that the computer treats you differently depending on your mood. Maybe to avoid getting a head through the screen.

Face First

A company called MindMaze are developing a product that may allow affective computing to come to VR headsets. Usually affective computing systems will use a camera to track facial expressions, but that’s a little difficult when the person in question has a big HMD strapped to their face.

The MASK by MindMaze is an HMD insert with electrodes built into the cushion. These measure the electrical activity of your facial muscles and interpret that into a digital representation of your facial expression.

According to MindMaze the system can detect your facial expression so quickly that it actually knows what face you are making before you’re making it. Since it already registers the muscle movement when they are first triggered. By the time the facial expression is actually on your face the MASK already sent it to the computer.

The company claims that their secret-sauce algorithms don’t require any training. You just put the MASK on your face and it knows how you’re feeling.

Going Social

It’s not just computers that can benefit from knowing what your current feelings are. People will be interacting with each other in VR as well. When we Skype with each other we can see faces clearly, but in VR we’re represented by avatars.

With something like the MASK the avatar’s face could accurately reflect your own without any special effort from you. It could make hanging out in VR with your friends a far more human place to be.

My Face is Tired

Based on what people who have tried the MASK have to say it seems that the technology is not yet as polished as you want a final consumer device to be. Fitting the MASK in between your HMD and face seems a little fiddly and the electrodes still have to be swabbed with a conductive gel, which is kind of gross to be honest.

The potential of the tech cannot be denied however and we’re excited to see where this idea will take VR in the future.