Title: AudioLM: a Language Modeling Approach to Audio Generation
Authors: Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, Neil Zeghidour
Published: 7th September 2022 (Wednesday) @ 13:40:08
Link: http://arxiv.org/abs/2209.03143v1
Abstract
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen speakers. Furthermore, we demonstrate how our approach extends beyond speech by generating coherent piano music continuations, despite being trained without any symbolic representation of music.
AudioLM Notes
- Relies on SoundStream tokens as a low-sampling rate target for a language modelling pretext task
- Measures phonetic discriminability via the “ABX” error rate metric - see Evaluating speech features with the Minimal-Pair ABX task: Analysis of the classical MFC/PLP pipeline
- with trigrams, measures how often a random instance a trigram (“bit”) is closer to another trigram (“bet”) instead of another occurrence of the same trigram (“bit”2)
- consider within-speaker and across-speaker ABX: when A and X are from same and different speakers resp.