Model Overview

Description:

DLPP (Deep-Learning Post-Processing) is a family of lightweight deep-learning networks trained for video post-processing. The model's introduction can be found at https://blogs.nvidia.com/blog/rtx-video-super-resolution/. To the public, it is also known as RTX VSR.

Post-processing techniques are used to improve the video quality for the end user. It is usually out of the video encoder and decoder loop so that the engineers can develop and improve it rapidly. Post-processing includes but is not limited to removing encoding artifacts, increasing the resolution, and increasing the frame rate. It has been proven that deep learning models can provide better video quality compared to traditional algorithms.

License/Terms of Use:

DLPP model license in NVIDIA GPU driver: https://www.nvidia.com/en-us/drivers/geforce-license/.
DLPP model license in RTX video SDK: https://developer.nvidia.com/downloads/rtx/sdk/rtx_video_sdk_v1.1.0.zip.

Deployment Geography:

Global.

Use Case:

Individual users who own an NVIDIA GPU can use it to improve their video quality.

Release Date:

Build.Nvidia.com 01/30/2023 via https://www.nvidia.com/en-us/drivers/
Build.Nvidia.com 06/04/2024 via https://developer.nvidia.com/rtx-video-sdk/

References:

1. Pixel Perfect: RTX Video Super Resolution Now Available for GeForce RTX 40 and 30 Series GPUs
2. RTX Video SDK

Model Architecture:

Architecture Type: Convolution Neural Network (CNN)

Network Architecture: U-Net-based CNN architecture

Computational Load:

Cumulative Compute:

• < 1*10^23

Estimated Energy and Emissions for Model Training:

• Total kWh = 150

• Total Emissions (tCO2e) = 0.62

Input(s):

Input Type(s):

• Image
  

Input Format(s):

• Image: Red, Green, Blue (RGB)
  

Input Parameters:

• Image: Two-Dimensional (2D)
  

Other Properties Related to Input:

• The input consists of a single RGB image. The model is designed to work primarily with 8-bit and 10-bit input formats. As a pre-processing step, the image must be normalized to the range [0,1]. The supported input resolution is 640x360 to 3840x2160. The input does not include an alpha channel (no transparency) and is represented in [R, G, B] format.
  

Output(s):

Output Type(s):

• Image
  

Output Format(s):

• Image: Red, Green, Blue (RGB)
  

Output Parameters:

• Image: Two-Dimensional (2D)
  

Other Properties Related to Output:

• The output consists of a single RGB image, ranging from 0 to 1. The output does not include an alpha channel (no transparency) and is represented in [R, G, B] format.
  

Software Integration:

Runtime Engine(s):

• CASK SDK.

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Turing
• NVIDIA Ampere
• NVIDIA Lovelace
• NVIDIA Blackwell

[Preferred/Supported] Operating System(s):

• Windows
• Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

This AI model can be embedded as an Application Programming Interface (API) call into the software environment described above.

Model Version(s):

• Version 1.0: First official release, which does super-resolution for videos played in Chrome and Edge browsers.
• Version 1.5: Improve the training data and updated the training flow to improve output image quality.
• Version 2.0: Added support for quantized CUDA kernels and 10-bit HDR input.

Training, Testing, and Evaluation Datasets:

Training Dataset:

We used internally captured camera footage and gaming recordings to train the model. The captured footage consists of diverse visual scenarios and various popular games.

Data Modality:

• Image

Video Training Data Size:

• Less than 10,000 Hours

Data Collection Method by dataset:

• Human

Labeling Method by dataset:

• Not Applicable

Properties:

We captured high-quality camera footage in the wild. The capture resolution is either 4K or over 4K, and does not contain any faces of passersby or anyone close-up. We have also collected high-quality, 4K recordings of popular games in the market. No personal data or confidential data is involved.

Dataset License(s):

• N/A

Testing Dataset:

Data Collection Method by dataset:

• Human

Labeling Method by dataset:

• Not Applicable

Properties:

• Same as the training dataset.

Dataset License(s):

• N/A

Evaluation Dataset:

Evaluation Result:

The DLPP model is evaluated mainly based on subjective quality assessment, in which engineers are invited to rate the output video quality. Their ratings are collected and averaged to compute the Mean Opinion Score (MOS). A Non-reference video quality evaluation metric NIQE is also employed. The evaluation results on the evaluation dataset are presented below.

Data Collection Method by dataset

• Human

Labeling Method by dataset:

• Not Applicable

Properties:

• A few dozen popular YouTube videos are used for evaluation.

Dataset License(s):

• N/A

Inference:

Engine: CASK SDK
Test Hardware:

• RTX 20 series
• RTX 30 series
• RTX 40 series
• RTX 50 series

Realtime Inference Latency:

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards.

Please report security vulnerabilities or NVIDIA AI Concerns here.