Google works on ‘removing headsets’ in mixed reality

Google is working on ways to reveal a person’s facial expressions when they are wearing a virtual reality headset.

Using a combination of machine learning, 3D computer vision, and advanced rendering techniques, Google claims it can now “remove” headsets to show a person’s identity, focus and full face in mixed reality.

Mixed reality is a way to convey what’s happening inside and outside a virtual place in a two dimensional format. With this new technology, we’re able to make a more complete picture of the person in VR,” said Google Research software engineer, Avneesh Sud, in a company blog post.

“Using a calibrated VR setup including a headset (like the HTC Vive), a green screen, and a video camera, combined with accurate tracking and segmentation, you can see the ‘real world’ and the interactive virtual elements together.”

By mimicking where a person is looking, even though it’s hidden under a headset Google has been able to create what it described as a “translucent ‘scuba mask’ look”.

Google can then calibrate the Vive tracking system and the external camera to create a vision of the person in the virtual world.

“The end result is a complete view of both the virtual world and the person in it, including their entire face and where they’re looking,” said Sud.

Google’s work on mixed reality is a joint effort between Google Research, Daydream Labs, and the YouTube team.