Categories: BlogsTechnology

Meta update: Introducing AI Models That Understand How the World Around Us Sounds

Meta Blog posted yesterday, “Today, our artificial intelligence (AI) researchers and audio specialists from our Reality Labs team, in collaboration with researchers from the University of Texas at Austin, are making three new models for audio-visual understanding open to developers. These models, which focus on human speech and sounds in video,  are designed to push us toward a more immersive reality at a faster rate. “

Whether it’s mingling at a party in the metaverse or watching a home movie in your living room through augmented reality (AR) glasses, acoustics play a role in how these moments will be experienced. We’re building for mixed reality and virtual reality experiences like these, and we believe AI will be core to delivering realistic sound quality. 

All three models tie into our AI research around audio-visual perception. We envision a future where people can put on AR glasses and relive a holographic memory that looks and sounds the exact way they experienced it from their vantage point or feel immersed by not just the graphics but also the sounds as they play games in a virtual world. 

These models are bringing us even closer to the multimodal, immersive experiences we want to build in the future.

Visual-Acoustic Matching

Anyone who has watched a video where the audio isn’t consistent with the scene knows how disruptive this can feel to human perception. However, getting audio and video from different environments to match has previously been a challenge. 

To address this, we created a self-supervised Visual-Acoustic Matching model, called AViTAR, which adjusts audio to match the space of a target image. The self-supervised training objective learns acoustic matching from in-the-wild web videos, despite their lack of acoustically mismatched audio and unlabeled data.

One future use case we are interested in involves reliving past memories. Imagine being able to put on a pair of AR glasses and see an object with the option to play a memory associated with it, such as picking up a tutu and seeing a hologram of your child’s ballet recital. The audio strips away reverberation and makes the memory sound just like the time you experienced it, sitting in your exact seat in the audience.

Also Read: Instagram extends Reels API Access to Third-Party Platforms

VisualVoice

VisualVoice learns in a way that’s similar to how people master new skills — multimodally — by learning visual and auditory cues from unlabeled videos to achieve audio-visual speech separation. 

For example, imagine being able to attend a group meeting in the metaverse with colleagues from around the world, but instead of people having fewer conversations and talking over one another, the reverberation and acoustics would adjust accordingly as they moved around the virtual space and joined smaller groups. VisualVoice generalizes well to challenging real-world videos of diverse scenarios.

Social Nation
Social Nation

Leave a Comment

Recent Posts

Zivame Launches Its New Campaign: ‘The Museum of Boobs’

National, November 18th, 2024: Zivame, India’s leading intimate wear brand, continues to reshape the intimate…

16 hours ago

Gurpreet Singh Bhasin: Pioneering The Creator Economy With Passion

Born entrepreneurs are a different breed. Imagine launching a business at 18! Gurpreet Singh Bhasin,…

3 days ago

What It Takes To Be Shabir Momin: The Visionary Disruptor Balancing Innovation And Leadership

Who gets to call themselves a Chief Technology Officer (CTO) at the age of 21?!…

3 days ago

Tata Sampann Launches New ‘Pyar Bhara’ Campaign With Chefs Sanjeev Kapoor And Ranveer Brar

~ To highlight how Tata Sampann brings to you food, with its goodness, as intended…

6 days ago

Tira Unveils Its Flagship Luxury Beauty Store At Jio World Plaza

Unveiling a New Era of Luxury Beauty: Tira redefines beauty retail in India with its…

7 days ago

From The Creator-Verse: Celebrating Big Milestones

The world of social media is dominated by content creators today. They have revolutionised the way…

1 week ago