Model Overview

This model card contains a NeMo Audio Codec model trained on the Libri-Light audiobook recordings dataset, comprising approximately 60,000 hours of English language speech with a 16kHz sampling rate. An Audio Codec model offers a low bit rate discrete representation of audio. The current 14M parameter model encodes audio information at a bitrate of 6.4kbps.

Model Architecture

The NeMo Audio Codec model is a non-autoregressive convolutional encoder-quantizer-decoder model for audio codec extraction. This 14M parameter model is trained end-to-end using time-domain loss, discriminative loss, and frequency domain loss, similar to other audio codec works Encodec [3].

Figure: NeMo Audio Codec Model Architecture

Training

The NeMo Audio Codec Model was trained using the NVIDIA NeMo [4] toolkit for 130k steps with an effective batch size of 256 on a single node of 8 NVIDIA V100s. The current model can be trained using this example script and this base configuration.

Datasets

The NeMo Audio Codec model is trained on over 60,000 hours of LibriVox audio recordings, also known as the Libri-Light dataset [5], with a sampling frequency of 16kHz. All recordings in this dataset are in the English language.

Performance

We assess NeMo Audio Codec model performance on various datasets to gauge the semantic and perceptual quality of the reconstructed audio produced by discrete tokens.

Perceptual audio quality of the reconstructed audio is evaluated using virtual speech quality objective listener metric (VISQOL) [2], and signal reconstruction is evaluated using scale-invariant signal-to-distortion ratio (SI-SDR) [1]. VISQOL, an objective full-reference metric for perceived audio quality, is assessed in audio mode at 48kHz and speech mode at 16kHz. SI-SDR evaluates signal reconstruction in terms of relative energy of the distortion in the reconstructed signal, accounting for scale or magnitude variations.

We evaluated audio codec model performance on three scenarios:

1. Evaluating Semantic content reconstruction on reconstructed audio from codecs Using a pretrained stt_en_fastconformer_ctc_large model
2. Evaluating Speaker Voice retaining ability of these models using TitaNet-L model
3. Training an ASR model with these discrete tokens and evaluating Word Error Rate (WER).

Evaluating Semantic Content Reconstruction

Table: NeMo Audio Codec Model performance is evaluated on Librispeech test clean set using various perceptual quality evaluations and reconstructed audio WER is also shown.

Table: NeMo Audio Codec Model performance is evaluated on Librispeech test other set using various perceptual quality evaluations and reconstructed audio WER is also shown.

Table: Performance evaluation on additional evaluation datasets with reconstructed audio. Here original audio model corresponds to evaluating on original audio with stt_en_fastconformer_ctc_large model.

Evaluating Speaker Voice Retaining Ability

Table Speaker Verification EER on voxceleb-test clean set on reconstructed audio and original audio using TitaNet-L model.

Evaluating Codes for Downstream Tasks

To evaluate performance of audio codecs on downstream tasks, we trained stt_en_fastconformer_rnnt_large on these audio codes/tokens for speaker recognition task.

How to Use this Model

The model is available for use in the NeMo toolkit Cite NeMo, and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.

Automatically load the model from NGC

from  nemo.collections.tts.models  import  AudioCodecModel
nemo_codec_model  =  AudioCodecModel.from_pretrained('audio_codec_16khz_small')

Getting discrete tokens from Audio

import librosa
import torch
audio,sr = librosa.load("<path_to_audio>", sr=16000)
device = 'cuda' if torch.cuda.is_available() else 'cpu'
audio_tensor = torch.from_numpy(audio).unsqueeze(dim=0).to(device)
audio_len = torch.tensor([audio_tensor[0].shape[0]]).to(device)
nemo_codec_model.freeze()
encoded_tokens, encoded_len = nemo_codec_model.encode(audio=audio_tensor, audio_len=audio_len)

Reconstructing audio from discrete tokens

import soundfile as sf
reconstructed_audio = nemo_codec_model.decode(tokens=encoded_tokens, tokens_len=encoded_len)

Listen to audio

output_audio  =  reconstructed_audio.cpu().numpy()[0]
sf.write(<path_to_output_audio>,  output_audio,  samplerate=16000)

Input

This model accepts single-channel raw audio signal sampled at 16000 Hz as input.

Output

Model encodes audio to discrete tokens and also decodes them to reconstructed audio as shown in above code snippet.

References

1. J. Le Roux et al., SDR – Half-baked or Well Done?, Proc. ICASSP, 2019.
2. Chinen et al., ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric, Proc. Intl. Conf. on Quality of Multimedia Experience (QoMEX), 2020.
3. Défossez et al., High Fidelity Neural Audio Compression, 2022.
4. NVIDIA NeMo Toolkit
5. Libri-Light: A Benchmark for ASR with Limited or No Supervision