Audio2Face 2D

The Audio2Face 2D collection contains the required components to run the Maxine AR SDK Audio2Face 2D effect. Audio2Face 2D, also known as Speech Live Portrait, is a generative model to create facial animations using a portrait photo and a driving audio such that the mouth movement in the photo synchronizes with the speech in the provided audio.

This feature is ready for commercial use.

Technical Details

For detailed model architecture, specifications, training information, and ethical considerations, please visit the Audio2Face 2D Model Page. To find the model page, click on the Artifacts tab above, and then click on Audio2Face 2D under the "Model" section.

Installation

SDK

To install the Audio2Face 2D feature:

1. Download and set up the AR SDK Core for your operating system. To find the Core SDK, click on the Artifacts tab above, and then click on Maxine Augmented Reality SDK Core under the "Resource" section.

   • For Linux: Choose the tag <Version>_linux (example: 1.0.0.0_linux)
   • For Windows: Choose the tag <Version>_windows (example: 1.0.0.0_windows)

   Follow the instructions on the SDK collection page to set up the SDK.

2. Install the Audio2Face 2D feature, also known as Speech Live Portrait, using the appropriate script:

   For Windows:

   cd <ARSDK_root>\features
   .\install_feature.ps1 -gpu <gpu_compute_capability> -features nvarspeechliveportrait -version <version>
   

   Example for an Ada GPU such as RTX 4090, which has GPU architecture sm_89 / CUDA Compute Capability 8.9:

   .\install_feature.ps1 -gpu 89 -features nvarspeechliveportrait -version 1.0.0.0
   

   For Linux:

   cd /usr/local/ARSDK/features
   ./install_feature.sh -f nvarspeechliveportrait -v <version>
   

   Example:

   ./install_feature.sh -f nvarspeechliveportrait -v 1.0.0.0
   

   Notes:

   • Windows: GPU parameter is required (e.g., 75, 86, 89, 120). See NVIDIA CUDA GPUs for your GPU's compute capability.
   • Linux: GPU is auto-detected, or use -g <gpu_type> (e.g., a100, t4, h100)
   • Replace <version> with your SDK version (default: 1.0.0.0)

The installation script will:

• Download the Audio2Face 2D feature libraries from NGC
• Download the appropriate TensorRT models for your GPU architecture
• Install them into the SDK's features directory

Try It Out

To try out the Audio2Face 2D feature, after downloading the SDK and feature, head over to the NVIDIA Maxine AR SDK Sample Apps GitHub repository to download and compile the sample applications that demonstrate the Audio2Face 2D functionality. This is demonstrated by the SpeechLivePortraitApp, and if using triton, by the SpeechLivePortraitTritonClientApp.

Documentation

User Guide: AR SDK Documentation

Terms of Use

The use of NVIDIA Maxine's Audio2Face 2D is governed by the following agreements:

• NVIDIA Software License Agreement
• Product Specific Terms for NVIDIA AI Products
• NVIDIA Community Model License

NIM Container

NVIDIA offers a ready-to-deploy NIM container for Audio2Face 2D:

Maxine Audio2Face 2D NIM

Audio2Face-2D animates a person's portrait photo using driving audio, creating realistic facial animations synchronized with speech.

To deploy the NIM container:

docker pull nvcr.io/nim/nvidia/maxine-audio2face-2d:latest

For detailed deployment instructions, visit the NIM container page.