Warning: Undefined array key "pathway" in /home/clients/160b20c93964292618c158d21ce27bf5/sites/tech.org-services.ch/wp-content/themes/Newspaper/functions.php on line 543
Monday, December 5, 2022
HomeArtificial Intelligencea Language Modeling Strategy to Audio Era

a Language Modeling Strategy to Audio Era

Producing practical audio requires modeling info represented at completely different scales. For instance, simply as music builds complicated musical phrases from particular person notes, speech combines temporally native constructions, akin to phonemes or syllables, into phrases and sentences. Creating well-structured and coherent audio sequences in any respect these scales is a problem that has been addressed by coupling audio with transcriptions that may information the generative course of, be it textual content transcripts for speech synthesis or MIDI representations for piano. Nevertheless, this strategy breaks when attempting to mannequin untranscribed elements of audio, akin to speaker traits essential to assist individuals with speech impairments recuperate their voice, or stylistic parts of a piano efficiency.

In “AudioLM: a Language Modeling Strategy to Audio Era”, we suggest a brand new framework for audio era that learns to generate practical speech and piano music by listening to audio solely. Audio generated by AudioLM demonstrates long-term consistency (e.g., syntax in speech, melody in music) and excessive constancy, outperforming earlier methods and pushing the frontiers of audio era with functions in speech synthesis or computer-assisted music. Following our AI Rules, we have additionally developed a mannequin to determine artificial audio generated by AudioLM.

From Textual content to Audio Language Fashions
In recent times, language fashions skilled on very giant textual content corpora have demonstrated their distinctive generative talents, from open-ended dialogue to machine translation and even commonsense reasoning. They’ve additional proven their capability to mannequin different indicators than texts, such as pure photographs. The important thing instinct behind AudioLM is to leverage such advances in language modeling to generate audio with out being skilled on annotated knowledge.

Nevertheless, some challenges must be addressed when shifting from textual content language fashions to audio language fashions. First, one should address the truth that the info fee for audio is considerably increased, thus resulting in for much longer sequences — whereas a written sentence will be represented by a couple of dozen characters, its audio waveform sometimes incorporates a whole lot of 1000’s of values. Second, there’s a one-to-many relationship between textual content and audio. Which means the identical sentence will be rendered by completely different audio system with completely different talking types, emotional content material and recording circumstances.

To beat each challenges, AudioLM leverages two sorts of audio tokens. First, semantic tokens are extracted from w2v-BERT, a self-supervised audio mannequin. These tokens seize each native dependencies (e.g., phonetics in speech, native melody in piano music) and world long-term construction (e.g., language syntax and semantic content material in speech, concord and rhythm in piano music), whereas closely downsampling the audio sign to permit for modeling lengthy sequences.

Nevertheless, audio reconstructed from these tokens demonstrates poor constancy. To beat this limitation, along with semantic tokens, we depend on acoustic tokens produced by a SoundStream neural codec, which seize the small print of the audio waveform (akin to speaker traits or recording circumstances) and permit for high-quality synthesis. Coaching a system to generate each semantic and acoustic tokens leads concurrently to excessive audio high quality and long-term consistency.


Coaching an Audio-Solely Language Mannequin
AudioLM is a pure audio mannequin that’s skilled with none textual content or symbolic illustration of music. AudioLM fashions an audio sequence hierarchically, from semantic tokens as much as fantastic acoustic tokens, by chaining a number of Transformer fashions, one for every stage. Every stage is skilled for the subsequent token prediction primarily based on previous tokens, as one would prepare a textual content language mannequin. The primary stage performs this activity on semantic tokens to mannequin the high-level construction of the audio sequence.


Within the second stage, we concatenate your complete semantic token sequence, together with the previous coarse acoustic tokens, and feed each as conditioning to the coarse acoustic mannequin, which then predicts the long run tokens. This step fashions acoustic properties akin to speaker traits in speech or timbre in music.


Within the third stage, we course of the coarse acoustic tokens with the fantastic acoustic mannequin, which provides much more element to the ultimate audio. Lastly, we feed acoustic tokens to the SoundStream decoder to reconstruct a waveform.


After coaching, one can situation AudioLM on a couple of seconds of audio, which allows it to generate constant continuation. With a purpose to showcase the final applicability of the AudioLM framework, we think about two duties from completely different audio domains:

  • Speech continuation, the place the mannequin is anticipated to retain the speaker traits, prosody and recording circumstances of the immediate whereas producing new content material that’s syntactically right and semantically constant.
  • Piano continuation, the place the mannequin is anticipated to generate piano music that’s coherent with the immediate by way of melody, concord and rhythm.

Within the video beneath, you’ll be able to take heed to examples the place the mannequin is requested to proceed both speech or music and generate new content material that was not seen throughout coaching. As you hear, observe that all the pieces you hear after the grey vertical line was generated by AudioLM and that the mannequin has by no means seen any textual content or musical transcription, however somewhat simply discovered from uncooked audio. We launch extra samples on this webpage.

To validate our outcomes, we requested human raters to take heed to brief audio clips and determine whether or not it’s an unique recording of human speech or an artificial continuation generated by AudioLM. Primarily based on the rankings collected, we noticed a 51.2% success fee, which isn’t statistically considerably completely different from the 50% success fee achieved when assigning labels at random. Which means speech generated by AudioLM is difficult to tell apart from actual speech for the common listener.

Our work on AudioLM is for analysis functions and we’ve got no plans to launch it extra broadly right now. In alignment with our AI Rules, we sought to grasp and mitigate the chance that folks might misread the brief speech samples synthesized by AudioLM as actual speech. For this goal, we skilled a classifier that may detect artificial speech generated by AudioLM with very excessive accuracy (98.6%). This exhibits that regardless of being (virtually) indistinguishable to some listeners, continuations generated by AudioLM are very simple to detect with a easy audio classifier. This can be a essential first step to assist shield in opposition to the potential misuse of AudioLM, with future efforts probably exploring applied sciences akin to audio “watermarking”.

We introduce AudioLM, a language modeling strategy to audio era that gives each long-term coherence and excessive audio high quality. Experiments on speech era present not solely that AudioLM can generate syntactically and semantically coherent speech with none textual content, but additionally that continuations produced by the mannequin are virtually indistinguishable from actual speech by people. Furthermore, AudioLM goes nicely past speech and may mannequin arbitrary audio indicators akin to piano music. This encourages the long run extensions to different forms of audio (e.g., multilingual speech, polyphonic music, and audio occasions) in addition to integrating AudioLM into an encoder-decoder framework for conditioned duties akin to text-to-speech or speech-to-speech translation.

The work described right here was authored by Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Olivier Teboul, David Grangier, Marco Tagliasacchi and Neil Zeghidour. We’re grateful for all discussions and suggestions on this work that we obtained from our colleagues at Google.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments