## Model Overview

### Description

Maxine Voice Font converts the speaker's voice in input audio to match the
target voice while keeping linguistic information and prosody (for example, rhythm and intonation) unchanged.

Voice Font is available under NVIDIA Maxine — a developer platform for deploying
AI features that enhance audio, video, and creating new experiences in real-time
audio-video communication. Maxine's state-of-the-art models create high-quality
AI effects using standard microphones and cameras without additional special
equipments.

NVIDIA Maxine is exclusively part of NVIDIA AI Enterprise for production
workflows — an extensive library of full-stack software, including AI solution
workflows, frameworks, pre-trained models, and infrastructure optimization.

### Terms of use

The use of NVIDIA Maxine's Voice Font is available as a demonstration of the
input and output of the Voice Font generative model. As such the user may select
one of the sample inputs and target voices and download the generated audio for
evaluation under the terms of the [NVIDIA MAXINE EVALUATION LICENSE AGREEMENT](https://developer.download.nvidia.com/maxine/nvidia-maxine-evaluation-license-24oct2023.pdf).

### References(s):

* [NVIDIA Maxine](https://developer.nvidia.com/maxine)

### Model Architecture

**Architecture Type:** Convolution Neural Networks (CNNs), Transformers,
Generative Adversarial Networks (GANs)  
**Network Architecture:** Encoder-Decoder  
**Model Version:** 1.0

### Input:

**Input Type(s):** Ordered List (audio samples for input audio), Target Voice <br>  
**Input Format(s):** FP32 (-1.0 to 1.0) <br>
**Other Properties Related to Input:** Pulse Code Modulation (PCM) audio samples with no encoding or pre-processing; 16kHz sampling rate required.

### Output:

**Output Type(s):** Ordered List (audio samples) <br>
**Output Format:** FP32 (-1.0 to 1.0) <br>   
**Other Properties Related to Output:** PCM audio samples with no encoding or post-processing; upsampled to 24kHz.

#### Software Integration

**Supported Hardware Platform(s):** Ada, Ampere, Turing  
**Test Hardware:** L40, A10, T10  
**Supported Operating System(s):** Linux

## Training & Evaluation

### Datasets

NVIDIA models are trained on a diverse set of public and proprietary datasets.
The Voice Font model is trained on a dataset that comprises over 6000 speakers
from diverse English accents and some European languages.

The public datasets used are listed below.

**Link:** [LibriTTS](https://www.openslr.org/60/)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
LibriTTS is a multi-speaker English corpus of approximately 585 hours of read
English speech, which is resampled at 16kHZ.

**Link:** [VCTK](https://datashare.ed.ac.uk/handle/10283/3443)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
This CSTR VCTK Corpus includes speech data uttered by 110 English speakers with
various accents. Each speaker reads out about 400 sentences, which were selected
from a newspaper, the rainbow passage and an elicitation paragraph used for the
speech accent archive.

**Link:** [DAPS](https://ccrma.stanford.edu/~gautham/Site/daps.html)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
The DAPS dataset has 15 versions of audio (3 professional versions and
12 consumer device/real-world environment combinations). Each version consists
of about 4.5 hours of data (about 14 minutes from each of 20 speakers).

**Link:** [Edinburgh 56 speaker dataset](https://datashare.ed.ac.uk/handle/10283/2791)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
Clean and noisy parallel speech database from 56 speakers designed to train and
test speech enhancement methods that operate at 48kHz.

**Link:** [Librispeech](https://openslr.org/12/),
[Multilingual Librispeech (MLS EN)](http://www.openslr.org/94/),
[LJSpeech](https://keithito.com/LJ-Speech-Dataset/),
Librivox English 22k dataset (from Evelina)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
Large-scale multilingual corpus derived from LibriVox audiobooks.
Librispeech comprises of 1000 hours of audio. MLS comprises of 8 languages,
including about 44.5K hours of English and a total of about 6K hours for other
languages. LJSpeech consists of 13,100 short audio clips of a single speaker
reading passages from 7 non-fiction books for a total of 24 hours. Librivox
English 22k dataset (from Evelina) is a privately curated dataset from librivox
audiobooks.

**Link:** [CREMA-D](https://www.kaggle.com/datasets/ejlok1/cremad)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
CREMA-D is a data set of 7,442 original clips from 91 actors. These clips were
from 48 male and 43 female actors between the ages of 20 and 74 coming from a
variety of races and ethnicities (African America, Asian, Caucasian, Hispanic,
and Unspecified). Actors spoke from a selection of 12 sentences. The sentences
were presented using one of six different emotions (Anger, Disgust, Fear, Happy,
Neutral, and Sad) and four different emotion levels (Low, Medium, High, and
Unspecified).

**Link:** [HUI-Audio-Corpus-German](https://opendata.iisys.de/dataset/hui-audio-corpus-german/)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
A high quality Text-To-Speech dataset created by researchers at IISYS. The
dataset contains several speakers. The 5 largest are listed individually, the
rest are summarized as other. All audio files have a sampling rate of 44.1kHz.

**Link:** [Crowdsourced high-quality Argentinian Spanish speech data set](https://openslr.org/61/),
[Crowdsourced high-quality Chilean Spanish speech data set](https://openslr.org/71/),
[Crowdsourced high-quality Colombian Spanish speech data set](https://openslr.org/72/),
[Crowdsourced high-quality Peruvian Spanish speech data set](https://openslr.org/73/),
[Crowdsourced high-quality Puerto Rico Spanish speech data set](https://openslr.org/74/),
[Crowdsourced high-quality Venezuelan Spanish speech data set](https://openslr.org/75/)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
Datasets which contains recordings of different dialects of Spanish including
5739 recordings of native speakers.

**Link:** [Crowdsourced high-quality Catalan speech data set](https://openslr.org/69/)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
Data set which contains recordings of Catalan.

**Link:** [Crowdsourced high-quality UK and Ireland English Dialect speech data set](https://www.openslr.org/83/)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
The dataset contains male and female high quality recordings of English from
various dialects of the UK and Ireland for a total of 17,877 lines.

**Link:** [Thorsten Müller (German Neutral-TTS dataset)](https://openslr.org/95/),
[Thorsten Müller (German Emotional-TTS dataset)](https://www.openslr.org/110/)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
Free regular and emotional single german speaker datasets by Thorsten Müller
(voice) and Dominik Kreutz (audio optimization) for tts training. The regular
dataset consists of more than 23 hours of audio and the emotional dataset
comprises of 300 identical phrases recorded in 8 different emotions (Neutral,
Disgusted, Angry, Amused, Surprised, Sleepy, Drunk, Whispering) for a total
of 175 minutes audio material.

**Link:** [SIWIS](https://datashare.ed.ac.uk/handle/10283/2353)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
The SIWIS French Speech Synthesis Database includes high quality French speech
recordings and associated text files, aimed at building TTS systems, investigate
multiple styles, and emphasis. A total of 9750 utterances from various sources
such as parliament debates and novels were uttered by a professional
French voice talent providing more than 10 hours of speech.

**Link:** [CSS10 French](https://www.kaggle.com/datasets/bryanpark/french-single-speaker-speech-dataset),
[CSS10 Spanish](https://www.kaggle.com/datasets/bryanpark/spanish-single-speaker-speech-dataset),
[CSS10 German](https://www.kaggle.com/datasets/bryanpark/german-single-speaker-speech-dataset)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
CSS10 is a collection of single speaker speech datasets for 10 languages. Each
of them consists of audio files recorded by a single volunteer and their aligned
text sourced from LibriVox. These datasets have approximately 19 hours of French,
24 hours of Spanish and 17 hours of German speech.

**Link:** [HiFi-TTS](https://www.openslr.org/109/)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
A multi-speaker English dataset for training text-to-speech models.
The Hi-Fi TTS dataset contains about 291.6 hours of speech from 10 speakers with
at least 17 hours per speaker sampled at 44.1 kHz.

**Link:** [JL-corpus](https://www.kaggle.com/tli725/jl-corpus)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
Emotional speech corpus with primary and secondary emotions. The corpus has
5 primary emotions (happy, angry, neutral, sad, and excited) and
5 secondary emotions (enthusiastic, apologetic, pensive, worried, and anxious).

**Link:** [PTDB-TUG](https://www.spsc.tugraz.at/databases-and-tools/ptdb-tug-pitch-tracking-database-from-graz-university-of-technology.html)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
The Pitch Tracking Database from Graz University of Technology (PTDB-TUG) is a
speech database for pitch tracking that provides microphone and laryngograph
signals of 20 English native speakers as well as the extracted pitch
trajectories as a reference. In total, this database consists of 4720 recorded
sentences.

**Link:** [Voxceleb 1 Dev data](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox1.html),
[Voxceleb 2 Dev data](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox2.html)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
VoxCeleb1 and VoxCeleb2 contain a total of over 1.1 million utterances for 1,251
and 6,112 celebrities, respectively, extracted from videos uploaded to YouTube.

**Link:** [RIR noise](https://www.openslr.org/28/)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
A database of simulated and real room impulse responses, isotropic and
point-source noises. The audio files in this data are all in 16KHz sampling rate
and 16-bit precision.

**Link:** Fisher ([Part-1 Speech](https://catalog.ldc.upenn.edu/LDC2004S13),
[Part-1 Transcripts](https://catalog.ldc.upenn.edu/LDC2004T19),
[Part-2 Speech](https://catalog.ldc.upenn.edu/LDC2005S13),
[Part-2 Transcripts](https://catalog.ldc.upenn.edu/LDC2005T19))  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
Fisher English Training Speech and Transcripts represent a collection of
conversational telephone speech (CTS) that was created at the LDC during 2003.
They contain a total of 11,699 audio files, each one containing a full
conversation of up to 10 minutes.

**Link:** [Switch board](https://catalog.ldc.upenn.edu/LDC97S62)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
The Switchboard Telephone Speech Corpus consists of approximately 260 hours of
speech and was originally collected by Texas Instruments in 1990-1, under DARPA
sponsorship.

**Link:** [WSJ-0](https://catalog.ldc.upenn.edu/LDC93S6A) and
[WSJ-1](https://catalog.ldc.upenn.edu/LDC94S13A)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
Corpus by the DARPA Spoken Language Program to support research on large-
vocabulary Continuous Speech Recognition (CSR) systems. The first two CSR
Corpora consist primarily of read speech with texts drawn from a machine-
readable corpus of Wall Street Journal news text and are thus often known as
WSJ0 and WSJ1. WSJ0 consists of 123 speakers while WSJ1 provides 275 speaker
with approximately 81 hours of speech.

**Link:** [National Speech Corpus (Part 1, Part 6)](https://www.imda.gov.sg/NationalSpeechCorpus)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
The National Speech Corpus (NSC) is the first large-scale Singapore English
corpus spearheaded by the Info-communications and Media Development Authority
(IMDA) of Singapore. It is approximately 1.2 TB in size split across 6 parts and
aims to become an important source of open speech data for automatic speech
recognition (ASR) research and speech-related applications.

**Link:** [VoxPopuli (EN)](https://github.com/facebookresearch/voxpopuli)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
A large-scale multilingual speech corpus for representation learning,
semi-supervised learning and interpretation. The corpus comprises of 400K hours
of unlabelled speech data for 23 languages and 1.8K hours of transcribed speech
data for 16 languages.

**Link:** [Europarl-ASR (EN)](https://www.mllp.upv.es/git-pub/ggarces/Europarl-ASR/)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
A 1300-hour English-language speech and text corpus of parliamentary debates for
(streaming) ASR training and benchmarking, speech data filtering and speech data
verbatimization.

**Link:** [Mozilla Common Voice (v7.0)](https://commonvoice.mozilla.org/en/datasets)  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
An open source, publicly available multi-language dataset of 75,879 voices that
anyone can use to train speech-enabled applications.

**Link:** GTC Dataset  
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  
A collection of talks from Nvidia GTC Conferences with a total of 103 speakers.

## Inference

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server)  
**Test Hardware:** L40, A10, T10

### Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established
policies and practices to enable development for a wide array of AI applications.
When downloaded or used in accordance with our terms of service, developers
should work with their supporting model team to ensure this model meets
requirements for the relevant industry and use case and addresses unforeseen
product misuse. For more detailed information on ethical considerations for this
model, please see the Model Card++ [Explainability](explainability), [Bias](bias),
[Safety & Security](safety-and-security), and [Privacy](privacy) Subcards. Please
report security vulnerabilities or NVIDIA AI Concerns
[here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

voice-font

Voice Font converts the speaker's voice in input audio to match the target voice while keeping linguistic information and prosody unchanged.