TAO Sparse4D Model Card

Model Overview

Description:

TAO Sparse4D is an advanced three-dimensional (3D) multi-camera detection and tracking network. It generates precise 3D bounding boxes and tracking IDs for a diverse set of objects across multiple camera views. The model was developed by NVIDIA as a part of the TAO model family. The provided model is pre-trained on the Multi-Target Multi-Camera (MTMC) Tracking 2025 subset from the NVIDIA PhysicalAI-SmartSpaces dataset also utilized for the 2025 AI City Challenge. This model is ready for commercial use.

The model in this card was trained & evaluated on the following warehouse-centric classes: Person, Fourier_GR1_T2_Humanoid, Agility_Digit_Humanoid, Nova_Carter, Transporter and Forklift.

License/Terms of Use:

GOVERNING TERMS: Use of this model is governed by the NVIDIA Open Model License.

Deployment Geography:

Global

Use Case:

TAO Sparse4D is designed for 3D multi-camera object detection and tracking in indoor environments like warehouses and logistics facilities. The model detects & tracks objects across multiple camera views for applications including warehouse automation, safety and workflow optimization in industrial settings, providing spatial understanding of a scene.

Release Date:

v1.0 Model - NGC - 06/13/2025

v1.1 Model - NGC - 11/30/2025

v2.0 Model - NGC - 01/23/2026 via NGC

v2.1 Model - NGC - 02/27/2026 via NGC

v2.2 Model (ResNet50) - NGC - 05/12/2026 via NGC

v2.2 Model (ResNet101) - NGC - 05/12/2026 via NGC

References

• Wang, Y., Pusegaonkar, S., Wang, Y., Li, A., Kumar, V., Sethi, C., ... & Biswas, S. (2026). A Unified 3D Object Perception Framework for Real-Time Outside-In Multi-Camera Systems. arXiv preprint arXiv:2601.10819.
• Lin, X., Pei, Z., Lin, T., Huang, L., & Su, Z. (2023). Sparse4d v3: Advancing end-to-end 3d detection and tracking. arXiv preprint arXiv:2311.11722.

Model Architecture

Architecture Type: Convolution Neural Network (CNN) based backbone with Transformer based decoder layers.

Network Architecture: ResNet101 backbone with Transformer based decoder layers.

Number of model parameters: 7.8 × 10⁷ parameters

This work leverages the Sparse4D v3 model & is tailored for indoor environments like warehouses with static camera setups. Sparse4D is a query based technique which involves sampling sparse features for better computational efficiency compared to other dense 3D detection & tracking based techniques. The architecture features a ResNet101 backbone and processes time-synchronized frames from multiple cameras. Key components include the backbone, a Feature Pyramid Network (FPN), and multiple decoder layers incorporating Multi-Scale Deformable Aggregation blocks (featuring key-point generation, feature sampling, visibility net module & embedding generation) along with refinement & classification layers. The model is trained on regression, classification & ID losses. The model also consists of an Instance Bank module utilized for tracking ID assignment & management. Along with the 3D bounding box information & object ID, Sparse4D also outputs an instance feature containing high-dimensional semantics from the image encoder. The training algorithm optimizes the network to minimize the classification loss, regression loss & ID loss.

Input:

Sparse4D is trained on RGB images, camera calibration files, ground truth 3D bounding boxes & object ids, and optional depth maps. Since performing 3D multi-camera detection and tracking across large regions like warehouses with high-density camera setups is computationally expensive, we partition large regions into overlapping groups called Bird's Eye View (BEV) group. Each BEV group contains multiple cameras and serves as the fundamental training unit for Sparse4D.

Input Type(s): Each BEV group will have cameras with RGB images, Camera calibration file, Ground Truth with 3D bounding boxes & Optional Depth Maps available via HuggingFace or generated synthetically by Nvidia Isaac Sim Replicator.

Input Format: Red, Green, Blue (RGB) images stored in raw png/jpg/hdf5 and Depth Maps stored in png/hdf5. Can support input resolution of 3 x 1080 x 1920 for both RGB & Depth Maps. Data preprocessing is required for grouping images & depth maps from multiple cameras to appropriate BEV groups. This can be done via the TAO Data Service.

Input Parameters: Multiple dimensions. See below for detailed model input shapes.

Other Properties Related to Input: 3 x 1080 x 1920 (C x H x W) resolution images for both RGB & Depth Maps. Data Pre-Processing needed via TAO Data Services. No alpha channel required.

Note that depth maps are optional and not required for evaluation or inference. The raw model inputs are as follows:

• No. of cameras is dynamic in the model.
• Model is initialized with zero tensors for the input_cached_feature, input_cached_anchor, prev_exists & interval_mask at the first frame. These tensors will be updated automatically via instance bank from the second frame & onwards.

Output:

Output Type(s): Tensors consisting of bounding boxes in 3D , object confidence scores, classes, class confidence scores, tracking object ID & instance features.

Output Format: List of Tensors.

Output Parameters: Multiple dimensions. See below for detailed model output shapes.

Other Properties Related to Output: Please see the details below.

The final output is a list with length of batch size after post-processing. Each element is a dictionary with the following keys:

where:

• x, y, z - 3D coordinates of the bounding-box centroid in the world coordinate system which is in meters.
• width, length, height - Box dimensions in meters along its x (width), y (length) and z (height) axes of the object-centered coordinate system, with the origin at the centroid.
• yaw - Euler angle in radians about the y-axis of the object-centered coordinate system defining the box’s heading in the world coordinate system. (Pitch and roll are assumed zero.)
• velocity - 3D velocity vector components in meters per second (m/s) in the world coordinate system, representing the object's instantaneous motion along the x, y, and z axes respectively.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s):

• DeepStream - 7.1
  
• TAO - 6.1.0
  

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Blackwell
  
• NVIDIA Hopper
  
• NVIDIA Lovelace
  
• NVIDIA Ampere
  

Preferred/Supported Operating Systems:

• Linux
  

These models are designed for use with NVIDIA hardware and software. For hardware, the models are compatible with any NVIDIA DataCenter GPU. For software, the models are specifically designed for the TAO Toolkit.

Model versions (Latest):

• trainable_v2.2 (ResNet50 / ResNet101) - Pre-trained model for Sparse4D.
• deployable_v2.2 (ResNet50 / ResNet101) - Model for Sparse4D deployable to DeepStream or TensorRT.

Training, Testing, and Evaluation Datasets

Dataset Overview

** Total Size:

** Total Number of Datasets: 3 Datasets in total including (MTMC Tracking 2025 and COSMOS augmented version of the MTMC Tracking 2025 dataset, SCOUT dataset)

** Dataset partition: Training 80%, Testing 10%, Validation 10%

** Time period for training data collection: Collected throughout the year 2024-2025.

** Time period for testing data collection Collected throughout the year 2024-2025.

** Time period for validation data collection Collected throughout the year 2024-2025.

The model is trained on the MTMC Tracking 2025 dataset available on HuggingFace.

Data Format & Preprocessing

Raw Data (MTMC Tracking 2025/AI City Challenge Format)

This raw data includes .mp4 RGB video files, depth maps stored in HDF5 format, ground_truth.json, calibration.json & map.png. For more details on these files including the ground truth format please refer to the raw dataset format here.

The raw data may also be obtained from Nvidia Isaac Sim Replicator Agent IRA. When collected via this route, RGB image data & depth maps can be in frame output/HDF5 output. Utilize the TAO Data Service to convert your raw data format (AICity) to pickle format (OVPKL) accordingly.

Pickle File Structure (OVPKL format)

The above raw data format needs to be converted to a pickle files for model training via the TAO DataService. The model will utilize these pickle files along with the above files for training. An example command can be found in the sparse4d finetuning notebook. The pickle file will consist of the items.

The process generates pickle files (.pkl) containing scene information. Each pickle file is a dictionary with the following top-level keys:

• metadata: (dict) Contains metadata about the dataset.
• infos: (list) A list of dictionaries, where each dictionary contains information about a specific frame in the scene.

metadata key

The metadata dictionary contains the following keys:

infos key

The infos key holds a list of dictionaries. Each dictionary in this list corresponds to a frame and contains the following keys:

cams Structure

The cams dictionary (within each element of the infos list) contains nested dictionaries, where each key is a camera identifier (e.g., 'Camera', 'Camera_01', etc.). Each of these camera-specific dictionaries has the following structure:

3D bounding box information is transformed from OV coordinates to BEV coordiantes for faster convergence during training. During inference, results are converted back to OV world coordinates.

Public Datasets

MTMC Tracking 2025 Subset from NVIDIA PhysicalAI-SmartSpaces dataset

Private Datasets

NVIDIA COSMOS-augmented version of the MTMC Tracking 2025 dataset was used for training to improve data diversity and robustness across varying scenarios.

Self-Sourced Synthetic Data

• Overall Size per Modality: 80% of the overall data used for training was synthetic data collected from NVIDIA Isaac Sim Replicator.
• A description of the methods used to synthetically generate training data: Various warehouse related assets were imported in a scene with virtual cameras recording image data.
  

Training Data

The dataset statistics used for pretraining the model are as follows:

• Data Modality - Image
  

• Image Training Data Size - 1 Million to 1 Billion Images
  

• Data Collection Method by dataset - Synthetic
  

• Labeling Method by dataset - Synthetic
  

Properties (Quantity, Dataset Descriptions, Sensor(s)):

Please see above.

• Model was trained on the following moving object classes: Person, Fourier_GR1_T2_Humanoid, Agility_Digit_Humanoid, Nova_Carter, Transporter and Forklift.
• Both validation & testing was conduction on random scenes from the MTMC Tracking 2025 subset.

Testing and Evaluation Datasets

Both validation & testing was conduction on random scenes from the MTMC Tracking 2025 subset.

• Data Collection Method by dataset - Synthetic
  

• Labeling Method by dataset - Synthetic
  

Properties (Quantity, Dataset Descriptions, Sensor(s)):
We utilize a random scenes from the test set of the MTMC Tracking 2025 subset.

Methodology and KPI

The key performance indicator is Higher Order Tracking Accuracy (HOTA) per-class and the average HOTA obtained across all classes. We utilize the HOTA evaluation methodology to assess multi-object tracking accuracy.

Higher Order Tracking Accuracy (HOTA) is a metric that balances detection accuracy and association accuracy into a single unified score. It decomposes tracking performance into Detection Accuracy (DetA), which measures how well objects are localized, and Association Accuracy (AssA), which measures how well detections are linked over time into consistent tracks. HOTA is computed as the geometric mean of DetA and AssA, averaged over a range of localization thresholds. This provides a balanced, holistic measure of tracking quality that captures both spatial accuracy and temporal consistency.

The following scores are for v2.2 models trained on MTMC Tracking 2025 subset. The evaluation set and training set is disjoint.

ResNet50 backbone (v2.2)

ResNet101 backbone (v2.2)

Real-time Inference Performance

Inference runs through the DeepStream pipeline on TensorRT with mixed precision (FP16+FP32). The TensorRT columns capture model-only latency, while the DeepStream columns add the instance-bank pre- and post-processing overhead. The table summarizes how many cameras each GPU supports at 30, 15, and 10 FPS for the v2.0 model. Numbers in bold are measured (all TensorRT values and certain DeepStream values). DeepStream values not in bold are estimates based on the guidance that the DS microservice adds approximately 30% overhead over Sparse4D TensorRT model performance.

Changelog

v2.2

• Released two backbone variants of the same recipe: ResNet50 (lower latency / smaller footprint) and ResNet101 (high accuracy), shipped under sparse4d_rn50 and sparse4d_rn101 NGC model cards respectively.
• The ResNet50 variant is built on top of an RT-DETR R50 pre-training initialization for stronger image features at half the backbone cost.
• Added pallet_truck to the trained class set (person, gr1_t2, agility_digit, nova_carter, transporter, forklift, pallet_truck).
• Expanded the training corpus to all available MTMC Tracking 2025 scenes (lazy-loaded), 3K+ BEV groups, and refreshed COSMOS-augmented data; mixed FPS training is unchanged.
• Refreshed kmeans anchor priors per backbone (_ov_kmeans900_v2.2_r50.npy, _ov_kmeans900_v2.2_r101.npy).

v2.1

• Trained on a significantly expanded dataset: more synthetic environments (retail, hospital, supermarket), GTC'24 warehouse data, COSMOS-transferred data with BEV groups, and real-world SCOUT data.
• Leverages the upgraded camera grouping algorithm (SDU v1.2).
• Corrected forklift orientation annotations.

v2.0

• Increased the number of BEV groups to 1337 to improve accuracy across all benchmarks.
• Added COSMOS-augmented datasets to the training set, using an 80:20 split between synthetic data and COSMOS augmentations.
• Introduced multiple FPS settings in training to better handle low-frame-rate streams (1, 2, 3, 5, 6, 10, 15, and 30 FPS).
• Increased the regression weight for rotation loss to improve bounding box fitting accuracy.

v1.1

• Updated input image resolution to 960×540 and introduced camera embeddings to lift maximum camera limits.
• Expanded training set to 34 scenes spanning 500+ cameras and 380 BEV groups (1–18 cameras each).
• Supported classes: Person, Fourier_GR1_T2_Humanoid, Agility_Digit_Humanoid, Nova_Carter, Transporter, Forklift.
• Mixed input frame rates (10, 15, 30 FPS) to improve robustness to lower streaming FPS.
• Enabled sparser camera grouping with random frame dropping to mirror Data Service frame-loss scenarios.
• Fixed rotation bug in object annotations and retrained the model with corrected labels.
• New MSDA FP16 Kernel which helps improve throughput by ~1.5x-2x for Data Center GPUs available in the DeepStream Pipeline inside the Nvidia Video Search and Summarization (VSS) - Warehouse Blueprint release.

v1.0

• First commercial warehouse-focused 3D detection and tracking release.
• Input image resolution: 1408×512 and cameras limited to 20.
• Supported classes: Person, Fourier_GR1_T2_Humanoid, Agility_Digit_Humanoid, Nova_Carter.

How to use this model

In order to use the model as pre-trained weights for transfer learning, please use the snippet below as a template for the model component of the experiment spec file to train a Sparse4D. For more information on experiment spec file, please refer to the Train Adapt Optimize (TAO) Toolkit User Guide.

model:
  type: "sparse4d"
  use_grid_mask: true
  use_deformable_func: true
  use_temporal_align: true
  input_shape: [960, 540]
  embed_dims: 256
  neck:
    type: "FPN"
    num_outs: 4
    start_level: 0
    out_channels: 256
    in_channels: [256, 512, 1024, 2048]
    add_extra_convs: "on_output"
    relu_before_extra_convs: true
  depth_branch:
    type: "dense_depth"
    embed_dims: "${model.embed_dims}"
    num_depth_layers: 3
    loss_weight: 0.2
  head:
    type: "sparse4d"
    num_output: 300
    cls_threshold_to_reg: 0.05
    decouple_attn: true
    return_feature: true
    use_reid_sampling: false
    embed_dims: "${model.embed_dims}"
    num_groups: 8
    num_decoder: 6
    num_single_frame_decoder: 1
    drop_out: 0.1
    temporal: true
    with_quality_estimation: true
    instance_bank:
      num_anchor: 900
      anchor: ???
      num_temp_instances: 600
      confidence_decay: 0.8
      feat_grad: false
      default_time_interval: 0.033333
      embed_dims: "${model.embed_dims}"
      use_temporal_align: "${model.use_temporal_align}"
    anchor_encoder:
      type: 'SparseBox3DEncoder'
      vel_dims: 3
      embed_dims: [128, 32, 32, 64]
      mode: 'cat'
      output_fc: false
      in_loops: 1
      out_loops: 4
    operation_order: [
      "deformable", "ffn", "norm", "refine", "temp_gnn", "gnn", "norm", 
      "deformable", "ffn", "norm", "refine", "temp_gnn", "gnn", "norm", 
      "deformable", "ffn", "norm", "refine", "temp_gnn", "gnn", "norm", 
      "deformable", "ffn", "norm", "refine", "temp_gnn", "gnn", "norm", 
      "deformable", "ffn", "norm", "refine", "temp_gnn", "gnn", "norm", 
      "deformable", "ffn", "norm", "refine"
    ]
    temp_graph_model:
      type: "MultiheadAttention"
      embed_dims: 512
      num_heads: 8
      batch_first: true
      dropout: 0.1
    graph_model:
      type: "MultiheadAttention"
      embed_dims: "${model.head.temp_graph_model.embed_dims}"
      num_heads: "${model.head.temp_graph_model.num_heads}"
      batch_first: true
      dropout: "${model.head.temp_graph_model.dropout}"
    norm_layer:
      type: "LN"
      normalized_shape: "${model.embed_dims}"
    ffn:
      type: "AsymmetricFFN"
      in_channels: 512
      pre_norm:
        type: "LN"
      embed_dims: 256
      feedforward_channels: 1024
      num_fcs: 2
      ffn_drop: 0.1
      act_cfg:
        type: "ReLU"
        inplace: true
    deformable_model:
      embed_dims: "${model.embed_dims}"
      num_groups: 8
      num_levels: 4
      attn_drop: 0.15
      use_deformable_func: true
      use_camera_embed: true
      residual_mode: "cat"
      kps_generator:
        embed_dims: "${model.embed_dims}"
        num_learnable_pts: 6
        fix_scale:
          - [0, 0, 0]
          - [0.45, 0, 0]
          - [-0.45, 0, 0]
          - [0, 0.45, 0]
          - [0, -0.45, 0]
          - [0, 0, 0.45]
          - [0, 0, -0.45]
    refine_layer:
      type: "SparseBox3DRefinementModule"
      embed_dims: "${model.embed_dims}"
      refine_yaw: true
      with_quality_estimation: true
    sampler:
      num_dn_groups: 5
      num_temp_dn_groups: 3
      dn_noise_scale: [2.0, 2.0, 2.0, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]
      max_dn_gt: 128
      add_neg_dn: true
      cls_weight: 2.0
      box_weight: 0.25
      reg_weights: [2.0, 2.0, 2.0, 0.5, 0.5, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0]
      use_temporal_align: "${model.use_temporal_align}"
    visibility_net:
      type: "visibility_net"
      embedding_dim: 256
      hidden_channels: 32
    loss:
      reg:
        type: "sparse_box_3d"
        box_weight: 0.25
        cls_allow_reverse: [5, 6, 7]
      cls:
        type: "focal"
        use_sigmoid: true
        gamma: 2.0
        alpha: 0.25
        loss_weight: 2.0
      id:
        type: "cross_entropy_label_smooth"
        num_ids: "${dataset.num_ids}"
    bnneck:
      type: "bnneck"
      feat_dim: 256
      num_ids: "${dataset.num_ids}"
    decoder:
      type: "SparseBox3DDecoder"
      score_threshold: 0.05
    reg_weights: [2.0, 2.0, 2.0, 1 ,1, 1, 1, 1, 1, 1, 1]

Inference & Deployment:

For deployment of this model, please refer to the our Nvidia Video Search and Summarization (VSS) - Warehouse Blueprint release documentation.

Acceleration Engine: Tensor(RT)

Test Hardware:

• NVIDIA Datacenter GPUs
  

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.

For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Sub-cards.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.