NVSaliENC Overview

Description:

NVSaliENC is an advanced perceptual video encoding module designed to improve visual quality in compressed video streams, specifically targeting critical regions as defined by human visual perception. While each generation of traditional video codecs provides measurable gains in objective quality at fixed bitrates, these improvements often do not correspond to noticeable visual differences for end users. Legacy codecs are limited in their ability to optimize perceptual quality, especially within the 2–5° visual angle around the viewer’s gaze center—a region where the human eye is most sensitive to detail.

NVSaliENC bridges this gap by employing a state-of-the-art deep learning-based saliency prediction model, expertly optimized for real-time inference using NVIDIA TensorRT. It is designed for seamless integration with NVIDIA NVENC, enabling practical and efficient deployment in production workflows. Unlike previous saliency solutions, NVSaliENC achieves real-time performance, making it suitable for latency-sensitive scenarios such as broadcasting, live streaming, and cloud gaming.

By generating high-precision saliency maps, NVSaliENC empowers NVENC to prioritize perceptual video compression for the most visually important regions, thereby enhancing end-user experience while reducing bandwidth usage and computational demand.

License/Terms of Use:

DLPP model license in NVIDIA GPU driver: https://www.nvidia.com/en-us/drivers/geforce-license/.

Deployment Geography:

Global

Use Case:

Individual users who own an NVIDIA GPU and a NVENC can use it to improve the video encoding visual quality.

Release Date:

NGC: 11/21/2025

References

NTECH 2025: NVSaliENC Human Vision-Guided Compression for Psycho-Visual Video Optimization

Model Architecture:

Architecture Type: Mixed Transformer-CNN based Network Architecture
Network Architecture:

The Mix-UNet network combines the strengths of transformer-based attention and multi-scale convolutional features to accurately capture both local and global visual cues relevant to human attention. We adopt a set of CNN decoders to extract spatial information from multi-scale features, and the residual connections help prevent any loss of information during the downsizing steps of the transformer encoders.

Number of model parameters: 5.2 * 10^7

Input:

Input Type(s): An image and an optional encoding mode
Input Format(s): YUV NV12, String
Input Parameters: Two-Dimensional (2D)
Other Properties Related to Input:

• Y channel: B X H X W X 1(Batch Size x Height x Width x Channel)
• UV channel: B X H/2 X W/2 X 2(Batch Size x Height/2 x Width/2 x Channel) in interleaved format.
• An optional input can be provided to specify the encoding use case: either VBR (variable bitrate) or CQ (constant quality).

Output:

Output Type(s): Image
Output Format: Delta QP map
Output Parameters: Two-Dimensional (2D)
Other Properties Related to Output: The output is a delta QP map in block format. Suppose the input resolution is H×W. The block size is 32 for H.265 and 64 for AV1. The dimensions of the delta QP map are (H // block_size, W // block_size). The value range is (-128, 127), where negative QP values allocate more bit resources to the target region.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems None. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), our model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s):

• TensorRT

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Turing
• NVIDIA Ampere
• NVIDIA Jetson
• NVIDIA Lovelace

[Preferred/Supported] Operating System(s):

• Linux
• Linux 4 Tegra

Model Version(s):

• Version 1.0: first stable release which supports vbr and cq mode. Requirement: TRT 10.14.

Training, Testing, and Evaluation Datasets:

Dataset Overview:

The datasets aim to enhance encoding quality around predicted salient regions, aligning with human vision system.

Training Dataset:

Link: NVSaliENC Datasets: NSPECT-4Q2T-2ZKV

Data Collection Method by dataset: Hybrid: Synthetic, Human.

Labeling Method by dataset:

• The general saliency annotations include fixations generated from mouse trajectories. To improve the data quality, isolated fixations with low local density have been excluded.
• The second plane saliency is synthesized by detecting the boxes containing texts.

Properties (Quantity, Dataset Descriptions, Sensor(s)): Resolution: 640x480.

Dataset License(s): NVSaliENC Datasets: https://jirasw.nvidia.com/browse/DGPTT-2487

Testing Dataset:

Link: NVSaliENC Datasets: NSPECT-4Q2T-2ZKV

Data Collection Method by dataset: Hybrid: Synthetic, Human.

Labeling Method by dataset:

• The general saliency annotations include fixations generated from mouse trajectories. To improve the data quality, isolated fixations with low local density have been excluded.
• The second plane saliency is synthesized by detecting the boxes containing texts.

Properties (Quantity, Dataset Descriptions, Sensor(s)): Resolution: 640x480.

Dataset License(s): NVSaliENC Datasets: https://jirasw.nvidia.com/browse/DGPTT-2487

Evaluation Dataset:

Data Collection Method by dataset:

• Human

Properties (Quantity, Dataset Descriptions, Sensor(s)): A few hundred videos from customers are used for evaluation.

Inference:

Acceleration Engine: TensorRT

Test Hardware:

• [NVIDIA A10]
• [NVIDIA T4]

Realtime Inference Latency:
The latency data was obtained when the input resolution was 3840x2160.

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards None.

Please report model quality, risk, security vulnerabilities or concerns NVIDIA AI Concerns here.