NVIDIA Audio2Emotion (A2E) is a deep learning model embedded within the Audio2Face-3D microservice that automatically recognizes emotions in human speech from audio input. The model analyzes vocal characteristics such as tone, pitch, and cadence to predict emotional states including joy, sadness, anger, fear, amazement, disgust, and others. These emotion predictions are used to drive the Audio2Face avatar's upper-face and eyebrow expressions, making the overall facial animation more natural and expressive beyond lip-sync alone. This release contains the Audio2Emotion v2.1 TensorRT engine, optimized for inference across supported NVIDIA GPU platforms.

Audio2Emotion Model Card

Model Overview

Description

This model is a speech emotion recognition (SER) classifier that can predict six emotions from speech: anger, disgust, fear, joy, neutral, and sadness. It is based on the Wav2Vec2 architecture and is trained to classify emotions in a sequence of audio frames.

This model is ready for commercial/non-commercial use.

License/Terms of Use

Use of this model is governed by the License Agreement for NVIDIA Audio2Emotion Model for Use with Audio2Face Project

AUDIO2EMOTION MODEL NOTICE: This model and any technology included with this model may only be used in connection with the NVIDIA Audio2Face project (https://docs.omniverse.nvidia.com/audio2face/latest/overview.html) consistent with all applicable documentation. You may not use this model and any technology included with it outside of the Audio2Face project. You may not use this model or any of its components for the purpose of emotion recognition.

Deployment Geography:

Global

Use Case:

IMPORTANT: This Model and any technology included with this Model may only be used in connection with the NVIDIA Audio2Face project (https://docs.omniverse.nvidia.com/audio2face/latest/overview.html) consistent with all applicable documentation. You may not use this Model and any technology included with it outside of the Audio2Emotion model outside the Audio2Face project. You may not use this Model or any of its components for the purpose of emotion recognition.

This speech emotion recognition model is specifically designed and optimized for the NVIDIA Audio2Face project to generate realistic facial expressions for 3D characters. The model's primary and intended use case is converting speech audio into emotional states that drive realistic 3D facial animations. The model is not intended for standalone emotion recognition applications or general-purpose audio analysis. It has been specifically trained and optimized to work as a component within the Audio2Face pipeline to produce high-quality, emotionally accurate 3D facial expressions that enhance the realism of virtual characters and digital humans.

Release Date:

Release Date: 08/27/2025 HuggingFace

---

Model Architecture

• Architecture Type: Transformer
• Network Architecture: Wav2Vec2
• This model was developed based on: Wav2Vec2-Large-LV60
• Number of model parameters: 3.1 x 10^8

Input

• Input Type(s): Audio
• Input Format(s): Raw audio input - an array of float32
• Input Parameters: 2D
• Other Properties Related to Input: A batch of input waveforms for classification

Output

• Output Type(s): Probabilities of emotional classes
• Output Format: An array of float32
• Output Parameters: 2D
• Other Properties Related to Output: The model can predict six emotions from speech: Anger, disgust, fear, joy, neutral, and sadness.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems [or name equivalent hardware preference]. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

---

Software Integration

Runtime Engine(s)

• NeMo - 1.0.0

Supported Hardware Microarchitecture Compatibility

• NVIDIA Ampere
  
• NVIDIA Blackwell
  
• NVIDIA Hopper
  
• NVIDIA Lovelace
  
• NVIDIA Pascal
  
• NVIDIA Turing
  

[Preferred/Supported] Operating System(s)

• Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
This AI model can be embedded as an Application Programming Interface (API) call into the software environment described above.

---

Model Version(s)

---

Training, Testing, and Evaluation Datasets

Training Dataset

Data Modality

• Audio

Audio Training Data Size

• Less than 10,000 Hours

Link

• Internal datasets
• RAVDESS
• CREMA-D
• JL Corpus
• EMO-DB
• Emozionalmente

Data Collection Method by dataset

• Automated

Labeling Method by dataset

• Human

Properties (Quantity, Dataset Descriptions, Sensor(s))

• Multiple datasets, including RAVDESS, CREMA-D, JL, EMO-DB, Emozionalmente, TTS GPT 4o (internal), Lindy & Rodney (internal)
• Quantity: 30029 samples

Testing Dataset

Link

• Internal dataset

Data Collection Method by dataset

• Automated

Labeling Method by dataset

• Human

Properties (Quantity, Dataset Descriptions, Sensor(s))

• Internal crowdsourced dataset
• Quantity: 1350 samples

Evaluation Dataset

Link

• Internal dataset

Data Collection Method by dataset

• Automated

Labeling Method by dataset

• Human

Properties (Quantity, Dataset Descriptions, Sensor(s))

• Internal crowdsourced dataset
• Quantity: 1350 samples

---

Inference

Engine

• Tensor(RT)

Test Hardware

• T4, T10, A10, A40, L4, L40S, A100
  
• RTX 6000ADA, A6000, Pro 6000 Blackwell
  
• RTX 3080, 3090, 4080, 4090, 5090
  

---

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, Explainability, Safety & Security, and Privacy Subcards.

Please report security vulnerabilities or NVIDIA AI Concerns here.

This Model and any technology included with this Model may only be used in connection with the NVIDIA Audio2Face project (https://docs.omniverse.nvidia.com/audio2face/latest/overview.html) consistent with all applicable documentation. You may not use this Model and any technology included with it outside of the Audio2Emotion model outside the Audio2Face project. You may not use this Model or any of its components for the purpose of emotion recognition.