## Model Overview

### Description:

Maxine Live Portrait generates facial animations using a portrait photo and a driving video such that the facial expressions and movements of the person in the video are transferred to the photo. Optionally, the speaker's voice in input video could also be transformed to match the target voice while keeping linguistic information and prosody (for example, rhythm and intonation) unchanged.

Maxine Live Portrait is deployed in NVIDIA Maxine, a developer platform for deploying AI features that enhance audio, video, and creating new experiences in real-time video communication. Maxine’s state-of-the-art models create high-quality AI effects using standard microphones and cameras without additional special equipments.

NVIDIA Maxine is exclusively part of NVIDIA AI Enterprise for production workflows—an extensive library of full-stack software, including AI solution workflows, frameworks, pre-trained models, and infrastructure optimization.

### Terms of use

The use of NVIDIA Maxine’s Live Portrait is available as a demonstration of the input and output of the Live Portrait generative model. As such the user may submit a reference “driving” video or use the sample “driving” video and download the generated video for evaluation under the terms of the [NVIDIA MAXINE EVALUATION LICENSE AGREEMENT](https://developer.download.nvidia.com/maxine/nvidia-maxine-evaluation-license-24oct2023.pdf).

### References(s):

* [NVIDIA Maxine](https://developer.nvidia.com/maxine)

### Model Architecture:

**Architecture Type:** Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs) 
**Network Architecture:** Encoder-Decoder 

### Input:

**Input Format:** RGBA image (portrait photo), RGB image (driving image), Target Voice 
**Input Parameters:** 720p to 4K 
**Other Properties Related to Input:** Input images pre-processed using proprietary technique; portrait photo supports 4 channel, 32 bit images; driving image supports 3 channel, 24 bit image. Pulse Code Modulation (PCM) audio samples with no encoding or pre-processing; 16kHz sampling rate required for transforming audio. 

### Output: 

**Output Format:** RGBA image 
**Output Parameters:** 512 x 512 or 1024 x 1024 (based upon configuration settings) 
**Other Properties Related to Output:** Input images post-processed using proprietary technique; 4 Channel, 32 bit image supported. PCM audio samples with no encoding or post-processing; upsampled to 24kHz is output for audio. 

### Supported Operating System(s):

Linux

### Model Version(s):

0.8.3

### Training and Evaluation Dataset:

Data Collection Method by dataset: Hybrid: Automated, Unknown 

Labeling Method by dataset: Hybrid: Automated, Unknown 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):**

Datasets used in live-portrait training are as follows:

One image dataset includes 7,356 facial images collected of 24 professional actors (12 female and 12 male) with different expressions, head poses, and backgrounds.

One image dataset includes a million videos of facial images of different expressions, head poses, and backgrounds.

The Voice Font model is trained on a dataset that comprises over 6000 speakers
from diverse English accents and some European languages.

#### Evaluation Dataset:

Data Collection Method by dataset: Human 
Labeling Method by dataset: Not Applicable 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):**

Internal, proprietary dataset of 65 facial images and 40 videos including 26 subjects of different ages, gender, and skin color with different expressions, head poses, and backgrounds.

NVIDIA models are trained on a diverse set of public and proprietary datasets. This model was trained on a dataset containing facial images of people covering different attributes such as expressions, head poses, backgrounds etc. NVIDIA is committed to the responsible development of AI Foundation models and conducts reviews of all datasets included in training.

## Inference:

**Engine:** [TensorRT](https://developer.nvidia.com/tensorrt), [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:**
* CUDA 11.8 compatible hardware versions.

### Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

live-portrait

Maxine Live Portrait is a generative model which animates a portrait photo with a driving video such that the facial expressions and movements of the person in the video are transferred to the photo.