Facebook has been going all in on augmented reality but the company isn’t just focused on what we see.
In a post, the social media firm peeled back the curtain on Facebook Reality Labs’ pioneering work developing audio-enhancing technology.
Their goal is to perfect ‘audio presence,’ where sounds from speakers, headphones or other devices are indistinguishable from a real-world source.
They’re also developing ‘perceptual superpowers,’ technology that tunes out background noise and let us focus on the person speaking to us by tracking the user’s eye movements.
Scroll down for video
Facebook Reality Labs is perfecting technology that would enable a phone call from a crowded bar to sound crystal clear. It’s also working to make sounds coming through speakers or headphones virtually identical to a real world source
Neither one is coming to market any time soon, but the technologies could have life-changing applications, including for those who are deaf or hard of hearing.
It’s likely the research will also finds its way into Facebook’s augmented reality glasses, expected to roll out some time between 2023 and 2025.
A top-flight team of researchers, engineers and designers are dedicated to enhancing what we hear, so that a conversation in a crowded cafe or a phone call from a thousands miles away both sound crystal clear.
By analyzing your eye and head movements, Facebook’s AR glasses with embedded speakers could identify what you’re trying to hear and dampen extraneous noise.
A prototype for Facebooks ‘perceptual superpowers’ technology, which tracks head and eye movements to tune out background noise and focus on the person speaking
Trying to listen to your date in a noisy bar? Stare into their eyes and the argument at the next table fades into a whisper.
The tech could also be incorporated with Facebook’s LiveMaps feature, with AI automatically screening out traffic and other distractions.
‘We are exploring new technologies that can extend, protect, and enhance your hearing ability — giving you the ability to increase concentration and focus, while allowing you to seamlessly interact with the people and information you care about,’ explains hardware research lead Tony Miller.
This isn’t just theoretical, as the lab has already developed prototypes that can track eye movements and adjust sounds accordingly.
Facebook Research Lab scientist Pablo Hoffmann (pictured) says spatial audio could be combined with ultra-realistic avatars and full-body tracking to create a virtual meeting indistinguishable from a face-to-face encounter
An always-on audio calibration system effectively lets users hear sounds in ultra-high fidelity through a pair of headphones with virtually no distortion or coloration.
The effect is that the recording is basically identical to the original, according to Oculus blog writer Lisa Brown Jaloza.
‘It’s so realistic, it’s virtually indistinguishable from the real thing,’ she writes after listening to research scientist Pablo Hoffman talk and then hearing the playback.
Facebook Research Labs’ in-ear audio monitors. While a lot of the work started out as part of AR and VR, says researcher Philip Robinson, ‘We realized that all of the technologies that we’re building here can serve a higher purpose, which is to improve human hearing’
‘In fact, sitting next to him at his desk, I’d bet money that he’s talking to me as I see him in my peripheral vision.’
Spatial audio could also be combined with ultra-realistic avatars and full-body tracking to create a virtual meeting indistinguishable from a face-to-face encounter.
‘Imagine if you were on a phone call and you forgot that you were separated by distance,’ researcher Philip Robinson says. ‘That’s the promise of the technology we’re developing.’
Robinson says while a lot of the work his team is doing started out as part of AR and VR, ‘We realized that all of the technologies that we’re building here can serve a higher purpose, which is to improve human hearing.’
In one experiment, Research scientist Vladimir Tourbabin calls Jaloza’s phone from a room playing loud music. Tourbabin flips a switch and his voice comes through clear as day.
‘It’s as if someone’s turned down the volume of the background noise so I can easily focus in on what he’s trying to say,’ she writes.
The key is a set of microphones on Tourbabin’s glasses that isolate his voice from the background noise.
‘The moment we’re in is such a crucial moment in the history of AR/VR technology,’ says Ravish Mehra, research science lead for FRL’s audio team.
‘Five or 10 years from now, if somebody joins the field, they’ll be chasing tail lights. At this moment, we can actually define what the future can be.’
Facebook has partnered with Ray-Ban parent company Luxottica on its AR glasses, CNBC reported last fall.
Codenamed Orion, the smart shades will replace smartphones, allowing users to make calls, view information on a small display, and live-stream their visual perspective to followers on social media.