Google DeepMind has introduced an innovative AI tool for generating video soundtracks. This advanced tool incorporates both text prompts and the contents of the video itself to create captivating and tailored soundtracks, and there’s also an optional text prompt that users can provide to describe the desired audio elements.
By combining these two elements, users can now produce scenes with “a drama score, realistic sound effects or dialogue that matches the characters and tone of a video.”
Examples of Soundtracks Created Using the AI Tool
To illustrate the capabilities of DeepMind's AI tool, several examples have been showcased on the company's website.
In one instance, a video featuring a car driving through a cyberpunk city was paired with a text prompt that included phrases such as "cars skidding," "car engine throttling," and "angelic electronic music." The generated audio synchronized perfectly with the car's movements, creating an exhilarating experience.
In another example, a prompt including terms like "jellyfish pulsating underwater," "marine life," and "ocean" produced an immersive underwater soundscape.
Other Features of this Soundtrack Generation Tool
The tool can autonomously generate an unlimited number of soundtracks for videos. This versatility makes it stand out from other AI tools in the market.
This variation in audio options allows users to explore a wide array of possibilities for their videos. It eliminates the need for meticulous audio synchronization, offering users endless creative opportunities.
DeepMind claims that its AI tool is trained on video, audio, and annotations with “detailed descriptions of sound and transcripts of spoken dialogue.” This comprehensive training enables the video-to-audio generator to perfectly link correspondingly the audio with visual scenarios.
DeepMind acknowledges the challenge of synchronizing lip movements with dialogue in its video-to-audio generation. The company is actively working on enhancing this aspect of the tool, aiming to achieve seamless synchronization between visual and auditory elements.
The quality of the input video can significantly affect the audio output. Grainy or distorted footage, according to DeepMind, may result in a significant decrease in audio quality. Users are encouraged to provide high-quality videos to ensure optimal audio synchronization and fidelity.
Comparison with Other AI Tools in the Industry
DeepMind's video soundtrack AI tool distinguishes itself from others in the industry, such as ElevenLabs' sound effects generator.
While ElevenLabs relies exclusively on text prompts, DeepMind's tool combines video pixels and text prompts, offering users a more comprehensive and immersive audio experience.
DeepMind's AI tool has the potential to enhance collaborations with other AI-generated video tools like Veo and Sora. These platforms can utilize DeepMind's tool to embed synchronized audio into their video generation capabilities, providing a more cohesive and immersive multimedia experience.
Google DeepMind includes a SynthID watermark in the AI-generated audio output for transparency and recognition purposes. This watermark serves as a flag, indicating that the audio has been generated using AI technology.