Categories: BlogsTechnology

Meta update: Introducing AI Models That Understand How the World Around Us Sounds

Meta Blog posted yesterday, “Today, our artificial intelligence (AI) researchers and audio specialists from our Reality Labs team, in collaboration with researchers from the University of Texas at Austin, are making three new models for audio-visual understanding open to developers. These models, which focus on human speech and sounds in video,  are designed to push us toward a more immersive reality at a faster rate. “

Whether it’s mingling at a party in the metaverse or watching a home movie in your living room through augmented reality (AR) glasses, acoustics play a role in how these moments will be experienced. We’re building for mixed reality and virtual reality experiences like these, and we believe AI will be core to delivering realistic sound quality. 

All three models tie into our AI research around audio-visual perception. We envision a future where people can put on AR glasses and relive a holographic memory that looks and sounds the exact way they experienced it from their vantage point or feel immersed by not just the graphics but also the sounds as they play games in a virtual world. 

These models are bringing us even closer to the multimodal, immersive experiences we want to build in the future.

Visual-Acoustic Matching

Anyone who has watched a video where the audio isn’t consistent with the scene knows how disruptive this can feel to human perception. However, getting audio and video from different environments to match has previously been a challenge. 

To address this, we created a self-supervised Visual-Acoustic Matching model, called AViTAR, which adjusts audio to match the space of a target image. The self-supervised training objective learns acoustic matching from in-the-wild web videos, despite their lack of acoustically mismatched audio and unlabeled data.

One future use case we are interested in involves reliving past memories. Imagine being able to put on a pair of AR glasses and see an object with the option to play a memory associated with it, such as picking up a tutu and seeing a hologram of your child’s ballet recital. The audio strips away reverberation and makes the memory sound just like the time you experienced it, sitting in your exact seat in the audience.

Also Read: Instagram extends Reels API Access to Third-Party Platforms

VisualVoice

VisualVoice learns in a way that’s similar to how people master new skills — multimodally — by learning visual and auditory cues from unlabeled videos to achieve audio-visual speech separation. 

For example, imagine being able to attend a group meeting in the metaverse with colleagues from around the world, but instead of people having fewer conversations and talking over one another, the reverberation and acoustics would adjust accordingly as they moved around the virtual space and joined smaller groups. VisualVoice generalizes well to challenging real-world videos of diverse scenarios.

Social Nation
Social Nation

Leave a Comment

Recent Posts

National Authors’ Day Special: Harshita Gupta a.k.a Soharshi On Her Debut Novel ‘Unloved’

Dealing with heartbreak, learning to let go and trying to move on has got to…

4 days ago

Revant Himatsingka A.K.A FoodPharmer: A Voice Brightening The Health World

Diwali celebrates the triumph of good over evil. On this special occasion, we wish to…

4 days ago

The Khooni Monday Horror Stories You Need To Hear This Halloween On The Social Nation Show

The hub of all things digital, Social Nation is on a quest to bring you real stories…

5 days ago

The Diwali Home Makeover Hacks You Need To Know Ft. Interior Designer Sonika Khurana Sethi

Didn't most of you spend the past weekend doing 'Diwali ki safai' at home? Do…

7 days ago

ZARA X Jayesh Sachdev New Collaboration ‘Reverie’ Is More Than An Artwork, It Is An Experience

To celebrate the joyful spirit of Diwali, Zara joins forces with the renowned multidisciplinary local…

1 week ago

Indian Actor and Content Creator Viraj Ghelani Goes International: Takes His Gujarati Stand Up Show To London and Leicester

Marking his standup debut in May, Viraj Ghelani is now all set to entertain the…

2 weeks ago