What Is NVIDIA NIM?

NVIDIA NIM™, part of NVIDIA AI Enterprise, is a set of easy-to-use microservices designed for secure, reliable deployment of high performance AI model inferencing across clouds, data centers and workstations. Supporting a wide range of AI models, including open-source and NVIDIA AI Foundation and custom models, it ensures seamless, scalable AI inferencing, on-premises or in the cloud, leveraging industry standard APIs.

Overview

NVIDIA Video Relighting NIM leverages state-of-the-art AI models to dynamically re-illuminate a person in a video with virtual studio lighting, enabling professional-quality lighting for video conferencing, content creation, and telepresence -- without requiring expensive physical lighting equipment. NVIDIA Video Relighting NIM models are built on the NVIDIA software platform, incorporating CUDA, TensorRT, and Triton to offer out-of-the-box GPU acceleration.

Architecture

NVIDIA Video Relighting operates as a multi-stage GPU-accelerated pipeline. First, the input video frame is segmented using the NVIDIA AI Green Screen model to produce a foreground matte that separates the subject from the background. The segmented foreground is then passed to the relighting neural network along with a High Dynamic Range Image (HDRI) environment map that defines the target lighting condition.

The relighting model analyzes the subject's geometry and appearance and renders re-illuminated output that matches the lighting environment specified by the HDRI. An equirectangular 360 x 180 degree projection of the HDRI onto a sphere provides the lighting information, and users can rotate the environment via a pan angle to change the direction of lighting. The model supports both a performance mode optimized for real-time applications and a quality mode for higher-fidelity post-production workflows.

The relit foreground is then composited over a background of the user's choice: the original source, a blurred version, the reprojected HDR environment, a custom image, or a solid color. The entire pipeline -- decode, segmentation, relighting, compositing, and encode -- runs on-GPU using NVIDIA DeepStream for hardware-accelerated video decoding and encoding via NVDEC/NVENC. Additional configurable parameters include foreground and background gain, specular highlight intensity, background blur strength, output resolution scaling, and five built-in HDR presets (Neutral, Cool, Cooler, Warm, Warmer) as well as support for custom HDR environment maps.

Input Processing Modes

NVIDIA Video Relighting NIM supports two distinct input processing modes:

• Streaming Mode: A specialized processing mode for streamable MP4 files where the input file is streamed to the NIM in chunks. As the chunks arrive, the NIM runs inference incrementally and streams the output back to the client, even before the entire input file has been received. This is the preferred mode for large files and low-latency applications.
• Transactional Mode: In transactional mode, the entire input video file is received and processed as a complete unit by the NIM before returning results to the client. This mode is suitable for smaller files or non-streamable inputs.

For detailed information about when to use each mode and how to convert between file formats, refer to the Input Modes section of Basic Inference.

Try It Out

Try the NVIDIA Video Relighting NIM at https://build.nvidia.com/nvidia/maxine-relighting

To experience the NVIDIA Video Relighting NIM API without having to host your own servers, use the Try API feature, which uses the NVIDIA Cloud Function backend.

Getting Started with NVIDIA NIM

Deploying and integrating NVIDIA NIM is straightforward thanks to our industry standard APIs. Visit the Maxine Relighting page for release documentation, deployment guides and more.

Get Help

Enterprise Support

Get access to knowledge base articles and support cases or submit a ticket.

Governing Terms

Downloadable: Use of the NIM container is governed by the NVIDIA Software License Agreement and Product-Specific Terms for NVIDIA AI Products. Use of this model is governed by the NVIDIA Open Model License.

API: This trial service is governed by the NVIDIA API Trial Terms of Service. Use of this model is governed by the NVIDIA Open Model License Agreement.

You are responsible for ensuring that your use of NVIDIA AI Foundation Models complies with all applicable laws.