NVIDIA Neural Reconstruction (NuRec) is a technology that converts recorded camera and lidar data into 3D scenes.
Overview
NVIDIA Neural Reconstruction (NuRec) converts recorded camera and lidar data into photorealistic 3D scenes suitable for simulation. Built on 3D Gaussian-based neural rendering, NuRec enables seamless real-to-sim workflows for training and testing Physical AI agents, including autonomous driving and robotics systems.
NuRec ingests multi-sensor captures — surround-view cameras and structured spinning lidar — and produces high-fidelity 3D reconstructions that can be rendered from novel viewpoints in real-time via a gRPC interface or exported as USDZ assets for use in NVIDIA Omniverse simulation platforms.
Key Features
- Neural 3D Reconstruction — Reconstruct static and dynamic scenes from multi-camera and lidar recordings using 3D Gaussian-based models.
- Real-Time Neural Rendering — Serve photorealistic novel-view renders through a built-in gRPC API.
- Multi-Sensor Fusion — Fuse data from multiple camera types and structured spinning lidar sensors.
- Scene Export — Export reconstructed scenes as USDZ/USDA assets, meshes, rig trajectories, and sequence tracks for downstream simulation and visualization.
- Multi-GPU Training — Distributed training support for scaling reconstruction across multiple GPUs and nodes.
- Harmonizer Enhancement — Optional generative post-processing with the Harmonizer diffusion model to harmonize color, lighting, and shadows, enforce temporal consistency, and remove reconstruction artifacts.
Use Cases
- Autonomous Vehicle Simulation — Convert real-world driving recordings into 3D environments for closed-loop AV testing and validation.
- Robotics (Real-to-Sim) — Create simulation-ready 3D scenes from physical environments to train and evaluate robotic agents.
- Sensor Simulation — Generate synthetic camera and lidar data from reconstructed scenes for sensor model development and testing.
- 3D Asset Generation — Reconstructed scenes can be used to extract individual 3D object assets for populating simulation worlds.
System Requirements
| Component | Requirement |
|---|---|
| GPU | NVIDIA GPU with CUDA compute capability; minimum 24 GB VRAM (48 GB+ recommended) |
| GPU Architectures | Ampere (A100, A10, A40, RTX A6000), Ada Lovelace (L20, L40, L40S), Grace Hopper (H20, H100), Blackwell (RTX Pro 6000D) |
| OS | Linux x86_64 |
| CUDA | 12.8 or higher |
| NVIDIA Driver | R560+ (R570+ recommended; R580+ for Blackwell) |
| Docker | 23.0.1 or later |
| NVIDIA Container Toolkit | 1.13.5 or later |
Supported Input Data
- Camera: images from pinhole, fisheye, and f-theta camera models; rolling and global shutter support; optional windshield distortion modeling.
- Lidar: Structured spinning lidar data (e.g., Hesai Pandar, AT128).
- Pose: Rig-to-world transforms with timestamps.
- Dataset Formats: NCore V3/V4
Data Handling
All processing occurs locally within the container. Recorded sensor data (camera images, lidar point clouds, poses) is read from and written to user-specified storage volumes. No data is transmitted externally by the container.
Documentation
Comprehensive documentation is available at https://docs.nvidia.com/nurec/
Governing Terms
Your use of the software and materials is governed by the NVIDIA Software License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and the Product-Specific Terms for NVIDIA Omniverse (found at https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-omniverse/). Additional Information: MIT License.