NVIDIA NIM for GPU accelerated Audio2Face-3D inference through gRPC APIs.
GOVERNING TERMS: Use of the ACE Audio2Face 3D NIM Container is governed by the NVIDIA Software License Agreement and Product-Specific Terms for NVIDIA AI Products. Use of the Audio2Face models is governed by the NVIDIA Open Model License. Use of the Audio2Emotion model is governed by the License Agreement for NVIDIA Audio2Emotion Model for Use with Audio2Face Project.
AUDIO2EMOTION MODEL NOTICE: This model and any technology included with this model may only be used in connection with the NVIDIA Audio2Face project (https://docs.omniverse.nvidia.com/audio2face/latest/overview.html) consistent with all applicable documentation. You may not use this model and any technology included with it outside of the Audio2Face project. You may not use this model or any of its components for the purpose of emotion recognition.
Audio2Face-3D NIM Overview
Audio2Face-3D NIM is a component of NVIDIA NIM™. NVIDIA NIM™ offers containers for self-hosting GPU-accelerated inferencing microservices, enabling deployment of pretrained and customized AI models across clouds, data centers, and workstations.
The Audio2Face-3D NIM converts speech into facial animation in the form of ARKit Blendshapes. The facial animation includes emotional expression. Where emotions can be detected, the facial animation system captures key poses and shapes to replicate character facial performance by automatically detecting emotions in the input audio. Additionally emotions can be directly specified as part of the input to the A2F-3D NIM. A rendering engine can consume Blendshape topology to display a 3D avatar’s performance.
Description:
The Audio2Face-3D (A2F-3D) microservice is a core component of our facial animation technology. It takes audio input and generates corresponding facial animations.
This NIM (NVIDIA Inference Microservice) is designed to simplify the deployment of optimized Audio2Face models, which excel in generating realistic facial animations from audio inputs for use in applications such as video conferencing, virtual reality, and digital content creation.
A2F-3D uses gRPC to integrate both client and server functionalities, facilitating seamless data streaming within the overall pipeline. It exposes the following gRPC endpoints:
- Bidirectional Streaming gRPC: For continuous processing of audio data and receiving animation data.
- Unary gRPC: For retrieving the current configuration of the microservice.
NVIDIA Audio2Emotion technology is integrated into Audio2Face, allowing for the automatic detection of emotions in human speech.
The container components are ready for non-commercial use.
Models used in the A2F-3D NIM container:
- Audio2Face-3D-v2.3 (opensourced)
- Audio2Face-3D-v3.0 (opensourced)
- Audio2Emotion-v2.2 (opensourced)
Deployment Geography: Global
Release Date:
NIM Release - NGC via https://catalog.ngc.nvidia.com/
Platform: x86/amd64
Model Release - Hugging Face: 08/27/2025 via
https://huggingface.co/nvidia/Audio2Face-3D-v2.3-Mark
https://huggingface.co/nvidia/Audio2Face-3D-v2.3.1-Claire
https://huggingface.co/nvidia/Audio2Face-3D-v2.3.1-James
https://huggingface.co/nvidia/Audio2Face-3D-v3.0
https://huggingface.co/nvidia/Audio2Emotion-v2.2
License/Terms of Use:
GOVERNING TERMS: Use of the ACE Audio2Face 3D NIM Container is governed by the NVIDIA Software License Agreement and Product-Specific Terms for NVIDIA AI Products. Use of the Audio2Face models is governed by the NVIDIA Open Model License. Use of the Audio2Emotion model is governed by the License Agreement for NVIDIA Audio2Emotion Model for Use with Audio2Face Project.
AUDIO2EMOTION MODEL NOTICE: This model and any technology included with this model may only be used in connection with the NVIDIA Audio2Face project consistent with all applicable documentation. You may not use this model and any technology included with it outside of the Audio2Face project. You may not use this model or any of its components for the purpose of emotion recognition.
You are responsible for ensuring that your use of NVIDIA provided models complies with all applicable laws.
Program Classes:
This NIM Container includes the following models:
| Model Name | Use Case | How to Pull the Model |
|---|---|---|
| Audio2Face-3D-v2.3 | Audio2Face-3D generates 3D facial animations from audio inputs, for use in applications such as video conferencing, virtual reality, and digital content creation. | Automatic. The pre-generated TensorRT Engines (based on these models) will be uploaded to https://catalog.ngc.nvidia.com/ and can be pulled via Automatic and Manual options. |
| https://huggingface.co/nvidia/Audio2Face-3D-v2.3-Mark | ||
| https://huggingface.co/nvidia/Audio2Face-3D-v2.3.1-Claire | ||
| https://huggingface.co/nvidia/Audio2Face-3D-v2.3.1-James | ||
| Audio2Face-3D-v3.0 | Audio2Face-3D generates 3D facial animations from audio inputs, for use in applications such as video conferencing, virtual reality, and digital content creation. | Automatic. The pre-generated TensorRT Engines (based on these models) will be uploaded to https://catalog.ngc.nvidia.com/ and can be pulled via Automatic and Manual options. |
| https://huggingface.co/nvidia/Audio2Face-3D-v3.0 | ||
| Audio2Emotion-v2.2 | Audio2Emotion model is a speech emotion recognition (SER) classifier that can predict six emotions from speech: anger, disgust, fear, joy, neutral, and sadness. It is based on the Wav2Vec2 architecture and is trained to classify emotions in a sequence of audio frames. | Automatic. The pre-generated TensorRT Engines (based on these models) will be uploaded to https://catalog.ngc.nvidia.com/ and can be pulled via Automatic and Manual options. |
| https://huggingface.co/nvidia/Audio2Emotion-v2.2 |
Deployment Details:
For more information on how to deploy, configure, and run the Audio2Face-3D NIM Docker container, please visit this link
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
Models can be deployed via:
- NIM container deployment
- Helm Chart
- Deployment on CSP (Azure, AWS) via the NV One Click scripts
Reference(s):
- Audio2Face-3D-Samples - GitHub
- Audio2Face-3D-v3.0 - Hugging Face
- Audio2Face-3D: Audio-driven Realistic Facial Animation For Digital Avatars
Container Version(s):
nvcr.io/nim/nvidia/audio2face-3d:2.0
Security Common Vulnerabilities and Exposures (CVEs)
Please review the Security Scanning tab on NGC to view the latest security scan results. For certain open-source vulnerabilities listed in the scan results, NVIDIA provides a response in the form of a Vulnerability Exploitability eXchange (VEX) document. The VEX information can be reviewed and downloaded from the Security Scanning tab.
Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal developer team to ensure these software components meet requirements for the relevant industry and use case and address unforeseen product misuse.
Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here.
Getting started with the NIM
Deploying and integrating the NIM is straightforward thanks to our industry standard APIs. Visit the NIM Container page for release documentation, deployment guides and more.