RT-DETR 2D Warehouse

Model Overview

Description:

The RT-DETR 2D Warehouse Perception Model v1.0 is part of NVIDIA’s RT-DETR family and features an EfficientViT/L2 backbone. It is pretrained on warehouse scene datasets for precise 2D object detection in industrial environments. This model is ready for commercial use.

The model in this card was trained & evaluated on the following warehouse-centric classes: Person, Fourier_GR1_T2_Humanoid, Agility_Digit_Humanoid, Nova_Carter, Transporter, Forklift and Pallet.

License/Terms of Use

GOVERNING TERMS: Use of this model is governed by the NVIDIA Open Model License.

Deployment Geography:

Global

Use Case:

Warehouse management personnel and logistics teams for 2D spatial object detection and tracking in AI-powered warehouse automation

Release Date:

NGC 01/30/2026 Model Link

References(s):

Zhao, Yian, et al. "Detrs beat yolos on real-time object detection." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.

Xie, Yulong, and Yu Liao. "Efficient-ViT: A light-weight classification model based on CNN and ViT." Proceedings of the 2023 6th International Conference on Image and Graphics Processing. 2023.

Model Architecture:

Architecture Type: Transformer

Network Architecture: RT-DETR w/ EfficientViT/L2 backbone

** This model is built upon Real-Time Detection Transformer (RT-DETR) architecture, incorporating the EfficientViT-L2 backbone to improve performance efficiency across edge and high-performance GPU systems. It retains RT-DETR’s end-to-end NMS-free detection mechanism while leveraging Vision Transformer (ViT) principles to efficiently balance speed and accuracy in large-scale object detection tasks.

The base model is PeopleNet Transformer v2.0. This model is fine-tuned from the pretrained base model.

RT-DETR processes the final three stages of the EfficientViT backbone (denoted as L0, L1 and L2) through an efficient hybrid encoder. This design combines Transformer-based global context modeling with CNN-style local spatial precision, leading to strong cross-scale generalization for small and dense object detection.

** Number of model parameters: 2.7*10^8

Input

Input Type(s): Image
Input Format: Red, Green, Blue (RGB)
Input Parameters: Four-Dimensional (4D) inputs
Other Properties Related to Input: Specific resolution: batch size x 3 x 544 x 960

The model expects a 4D tensor as input, representing a batch of RGB images. The tensor should have the shape [batch_size, C, H, W].

• batch_size: The number of images processed in a single forward pass.
• C (Channels): The number of color channels for the image. For standard RGB images, this value is 3.
• H (Height): The height of the input image in pixels (e.g., 544).
• W (Width): The width of the input image in pixels (e.g., 960).

The expected input format is batch_size x 3 x 544 x 960.

Output

Output Type(s): Numerical vectors
Output Format: Four-dimensional tensors
Output Parameters: Prediction Logits (pred_logits): Shape [1, batch size, number of queries, number of classes] (default: [1, batch size, 300, 7]). These represent the unnormalized prediction scores for each object query and class. Predicted Bounding Boxes (pred_boxes): Shape [1, batch size, number of queries, 4] (default: [1, batch size, 300, 4]). These correspond to the coordinates of the predicted bounding boxes for each object query.

The model returns a dictionary containing two primary tensors that hold the object detection predictions.

• Prediction Logits (pred_logits): This is a 4D tensor with a shape of [1, batch_size, num_queries, num_classes].

  • Default Shape: [1, batch_size, 300, 7]
  • Description: This tensor contains the unnormalized prediction scores (logits) for each object query. Each logit represents the model's confidence that a detected object belongs to a particular class. A higher value indicates a higher probability after applying a softmax function.

• Predicted Bounding Boxes (pred_boxes): This is a 4D tensor with a shape of [1, batch_size, num_queries, 4].

  • Default Shape: [1, batch_size, 300, 4]
  • Description: This tensor provides the coordinates for each predicted bounding box. The four values per box typically represent its center coordinates, width, and height (e.g., [center_x, center_y, width, height]), usually normalized relative to the image dimensions.

Other Properties Related to Output: Logit values are real-valued scores that represent the model’s level of confidence in each class per object query, prior to any normalization (e.g., softmax) or probability transformation. Bounding boxes contain four real numbers encoding the box location and dimensions per detected object.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s):

• DeepStream - 8.0
  
• TAO - 6.25.10+
  

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
  
• NVIDIA Blackwell
  
• NVIDIA Jetson
  
• NVIDIA Hopper
  
• NVIDIA Lovelace
  
• NVIDIA Pascal
  
• NVIDIA Turing
  
• NVIDIA Volta
  

Preferred/Supported Operating System(s):

• Linux 6.17.x kernel
  
• Linux 4 Tegra 35.4.x kernel
  

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

This AI model can be embedded as an Application Programming Interface (API) call into the software environment described above.

Model Version(s):

• deployed version: rtdetr_warehouse_v1.0.fp16.onnx
• trainable version: rtdetr_warehouse_v1.0.pth
  

Training, Testing, and Evaluation Datasets:

Dataset Overview

** Total Size: 109K images
** Total Number of Datasets: 6
** Modality: RGB image files
** Categories: 7 categories in total. Person, humanoid (Agility Digit, Fourier GR1_T2), Nova carter, transporter, forklift and pallet.

Training Datasets

AICity 2025: 2.9M synthetic data w/ annotations generated by NVIDIA IssacSim. A subset of AICity dataset is available at Physical AI Smart Spaces.

• Data Collection Method: Synthetic
• Labeling Method: Synthetic
• Properties:
  • Number of items: 9K images randomly sampled from 2.9M data.
  • Modalities: Images/Video

Cosmos AICity 2025: 2.7M of AICity data is further processed by NVIDIA Cosmos-Transfer1 for diverse world simulation. Cosmos AICity 2025 shares the same labels with AICity 2025.

• Data Collection Method: Synthetic
• Labeling Methos: Synthetic
• Properties:
  • Number of items: 8K images randomly sampled from 2.7M data.
  • Modalities: Images/Video

OpenImage V5 1.7M training images with 600 boxable object classes. Only person class annotations are used in RT-DETR Warehouse training.

• Data Collection Method: Automated (Filtered from existing dataset)
• Labeling Method: Automated (Pseudo-labeling)
• Properties:
  • Number of items: 5K randomly selected images from original dataset.
  • Modalities: Images

Astro2.2: Proprietary dataset a mix of cameras, camera heights, crowd-density, and field-of view (FOV) taken from a camera. Approximately half of the training data consisted of images captured in an indoor office environment. The camera is typically set up at approximately 10 feet height, 45-degree angle and has close field-of-view.

• Data Collection Method: Human
• Labeling Method: Human
• Properties:
  • Number of items: 20K radomly selected frames from 80K images.
  • Modalities: Video, Images

Warehouse Videos: 48 warehouse videos from publicly available internet scale data. 20K highlight frames are extracted from the videos and labeled by human.

• Data Collection Method: Human (Downloaded)

• Labeling Method: Human

• Properties:

  • Number of items: 38K highlight frames (from 48 videos)
  • Modalities: Video, Images

• Measures implemented to respect reservations of rights from the text and data-mining exception during data collection including specification of the opt-out protocols and solutions honored by the provider.
  

Roboflow Forklift Dataset: Downloaded 20 public datasets from Roboflow website.

• Data Collection Method: Human (Downloaded)
• Labeling Method: Automated (Pseudo-labeling)
• Properties:
  • Number of items: 29K images
  • Modalities: Images

Evaluation Datasets

** Building K: A proprietary dataset. It contains a set of outside-in 7 multi-view camera videos collected in 2025.

** Warehouse Videos Test: A proprietary dataset for evaluation.

• Data Collection Method: Human (Downloaded)
• Labeling Method: Human

** Warehouse Synthetic Test: A proprietary dataset. One scene with 8 outside-in multi-camera videos is from AICity 2025 for evaluation.

• Data Collection Method: Synthetic
• Labeling Method: Synthetic

Properties (Quantity, Dataset Descriptions, Sensor(s)): 3 evaluation datasets of images for 2D detection task.

Methodology and KPI

The key performance indicators are Average Precision (AP) per-class evaluated on the Warehouse Synthetic Test dataset. AP quantifies a detector's ability to trade off precision and recall for a single object category by computing the normalized area under its precision-recall curve.

The model supports 7 object categories: Person, Agility Digit (humanoid robot), Fourier GR1_T2 (humanoid robot), Nova Carter, Transporter, Forklift, and Pallet.

Evaluation Settings

The reported metrics use the following evaluation configuration:

• AP Variant: COCO AP@0.50
• IoU Thresholds: 0.50
• Max Detections: 100 detections per image
• Matching Policy: Greedy matching based on IoU with ground truth boxes, highest confidence predictions matched first

Real-time Inference Performance

Acceleration Engine: Tensor(RT)
Test Hardware:

• Jetson AGX Thor, JetPack SDK 7
  
• H100, Drive 580.65
  

RT-DETR + EfficientViT-L2 Model with supported No. of streams​

Inference Backend: TensorRT

How to use this model

In order to use the model as pre-trained weights for transfer learning, please use the snippet below as a template for the model component of the experiment spec file to train a Sparse4D. For more information on experiment spec file, please refer to the Train Adapt Optimize (TAO) Toolkit User Guide - RTDETR.

model:
  backbone: efficientvit_l2
  dec_layers: 6
  enc_layers: 1
  num_queries: 300
  return_interm_indices:
  - 1
  - 2
  - 3
  train_backbone: true

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.

For more detailed information on ethical considerations for this model, please see link for Model Card++ Bias, Explainability, Safety & Security, and Privacy Subcards.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.