TAO Pretrained Mask Grounding DINO with Non-Commercial License

Model Overview

Description

Open vocabulary instance segmentation is a computer vision technique that can segment one or multiple objects in a frame based on the text input. Object segmentation recognizes the individual objects in an image and predicts bounding boxes and the segmentation masks. This model card contains pre-trained weights for the Mask Grounding DINO model pretrained on the research dataset. The goal of this card is to facilitate transfer learning through the Train Adapt Optimize (TAO) Toolkit.

This model is ready for non-commercial use.

License

GOVERNING TERMS: Use of this model is governed by the NVIDIA License

Deployment Geography:

Global

Use Case:

1. Open-Set (or Open-Vocabulary) Object Detection and Segmentation: Grounding DINO can detect objects that were not pre-defined in the training categories — you can give it new category names via text (e.g., “lion”, “bench”, “ear”) and it will localise them in images. This is useful in systems that need flexibility to recognise arbitrary categories without retraining for each new class.

2. Referring Expression / Text-Guided Object Detection and Segmentation: The model supports being given a referring expression (“the bottom man with his head up”, “the dog next to the tree”) and then detect/localise the object described.

Release Date:

NGC 11/25/2025 via [URL]

References

• Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L., Shum, H.: DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection.
• Ziu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., Zhang, L.: Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.
• Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: PRe-training of Deep Bidirectional Transformers for Language Understanding.
• Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.
• Tian Z., Shen C., Chen H.: Conditional Convolutions for Instance Segmentation.
• Liu, C., Ding, H., & Jiang, X.: Gres: Generalized referring expression segmentation.

Model Architecture

Architecture Type: Transformer-Based Segmentation Model
Network Architecture: Swin-Tiny
Number of Parameters: 182,668,026 params

This model is an NVIDIA proprietary open-vocabulary object segmentation model that takes RGB images and a list of phrases or a referring expression as input, and produces bounding boxes, masks, and corresponding labels as output.

Specifically, the model was initialized from a research checkpoint of Mask Grounding DINO v1. It employs a Swin-Tiny backbone with a segmentation head pretrained end-to-end on the COCO and RefCOCO variant datasets, followed by fine-tuning of the ReLA module on approximately 200,000 image–referring expression pairs from the GRefCOCO dataset to enhance region-language grounding.

Input(s):

Input Type(s):
Image and Text (list of tokenized captions processed via Hugging Face).

Input Format(s):

• Image: Red, Green, Blue (RGB)
• Text: Tokenized string inputs
• Resolution: Minimum 32 × 32 pixels required
• Other: No alpha channel; standard 8-bit color depth
  

Input Parameters:

• Image: Two-Dimensional (2D)
• Text: One-Dimensional (1D)
  

Other Properties Related to Input:

• inputs: B × 3 × 544 × 960 — RGB image tensor (Batch Size × Channels × Height × Width)
• input_ids: B × 256 — tokenized text input IDs (Batch Size × Max Token Length)
• attention_mask: B × 256 — attention mask for text tokens
• position_ids: B × 256 — positional encoding indices
• token_type_ids: B × 256 — token type indicators for multi-sentence text
• text_token_mask: B × 256 × 256 — attention map between tokens (Batch Size × Max Token Length × Max Token Length)

Because ONNX and TensorRT do not support string inputs directly, the tokenization step is performed externally using the Hugging Face tokenizer.
For deployment details, refer to the TAO Deploy Repository.

Output(s):

Output Type(s):
Bounding Boxes, Confidence Scores, Segmentation Masks for each detected object in the input image, and a No-Target Indicator.

Output Format(s):

• Bounding Boxes: Predicted 2D boxes in (cx, cy, w, h) format
• Confidence Scores: Scalar scores for each predicted object
• Segmentation Masks: 3D masks aligned with the input image dimensions
• No-Target Indicator: Binary logits indicating whether a query corresponds to a valid object or not
  

Output Parameters:

• Bounding Boxes: Two-Dimensional (2D)
• Segmentation Masks: Three-Dimensional (3D)
• Confidence Scores: One-Dimensional (1D)
• No-Target Indicator: Two-Dimensional (B × 2)
  

Other Properties Related to Output:

• pred_logits: B × 900 — classification logits for each query (Batch Size × Number of Queries)
• pred_boxes: B × 900 × 4 — predicted bounding boxes in (cx, cy, w, h) format
• pred_masks: B × 900 × H × W — segmentation masks for each predicted object
• no_targets: B × 2 — binary logits indicating positive/negative (object/no-object) classes
• union_logit_masks: B × H × W — union of predicted object masks across all queries (objectness map)

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Hardware Optimization Note:
This model is optimized for NVIDIA GPU-accelerated systems.
By leveraging CUDA, TensorRT, and other NVIDIA frameworks, it achieves significantly faster training and inference performance compared to CPU-only systems.

Software Integration

Runtime Engines:

• TAO 5.5.0
  

Supported Hardware Architectures:

• NVIDIA Ampere
  
• NVIDIA Jetson
  
• NVIDIA Hopper
  
• NVIDIA Lovelace
  
• NVIDIA Pascal
  
• NVIDIA Turing
  
• NVIDIA Volta
  

Supported Operating Systems:

• Linux
  
• Linux 4 Tegra
  

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Versions

• mask_grounding_dino_swin_tiny_research_trainable_v2.0 - Pretrained Swin-Tiny Mask Grounding DINO model on gRefCOCO dataset.

Training and Evaluation

This model was trained using the mask_grounding_dino entrypoint in TAO. The training algorithm optimizes the network to minimize the localization and contrastive embedding loss between text and visual features as well as the mask losses between predictions and mask groundtruth.

Using this Model

These models need to be used with NVIDIA hardware and software. For hardware, the models can run on any NVIDIA GPU including NVIDIA Jetson devices. These models can only be used with Train Adapt Optimize (TAO) Toolkit, or TensorRT.

The intended use for these models is detecting objects in a color (RGB) image. The model can be used to detect objects from photos and videos by using appropriate video or image decoding and pre-processing.

These models are intended for training and fine-tune with the TAO Toolkit and your datasets for object detection. High-fidelity models can be trained with new use cases. A Jupyter Notebook is available as a part of the TAO container and can be used to re-train.

The models are also intended for easy edge deployment using TensorRT.

Using the Model with TAO

To use these models as pretrained weights for transfer learning, use the following snippet as a template for the model and train components of the experiment spec file. For more information on the experiment spec file, see the TAO Toolkit User Guide.

train:
  pretrained_model_path: /path/to/the/mask_groundingdino.pth
  freeze: ["backbone.0", "bert"]  # freeze the backbone for finetuning
model:
  backbone: swin_tiny_224_1k
  num_feature_levels: 4
  dec_layers: 6
  enc_layers: 6
  num_queries: 900
  dropout_ratio: 0.0
  dim_feedforward: 2048
  log_scale: auto
  class_embed_bias: True
  num_region_queries: 100
  loss_types: ['labels', 'boxes', 'masks', 'rela']

Training and Evaluation Datasets

Training Data

gRefCOCO

Link: https://github.com/henghuiding/gRefCOCO

Data Collection Method by dataset:
Human

Labeling Method by dataset:
Human

Properties:
Mask Grounding DINO was fine-tuned on the gRefCOCO dataset, which contains referring expressions paired with object segmentations. The dataset provides precise masks and bounding boxes for all target instances referenced in natural language expressions. gRefCOCO includes 19,994 images, covering 60,287 distinct object instances and a total of 278,232 referring expressions, of which 80,022 are multi-target and 32,202 are no-target expressions.
Part of the single-target expressions are inherited from RefCOCO. The dataset follows the UNC partition of RefCOCO for data splits and uses a quality-controlled annotation procedure inspired by ReferIt to ensure annotation consistency and accuracy.

Evaluation Data

Data Collection Method by dataset:
Human

Labeling Method by dataset:
Human

Properties (Evaluation Sets and General Information):

Referring-expression segmentation (RES) requires models to output object masks given natural language expressions. Classic datasets (RefCOCO, RefCOCO+, RefCOCOg) mostly focus on single-target expressions, while gRefCOCO supports single-, multi-, and no-target expressions, making it suitable for generalized RES (GRES) tasks.

• gRefCOCO:

  • Validation Set: 16,870 expressions / 8,163 sets / 1,500 images
  • Test Set A (People): 18,712 expressions / 6,266 sets / 750 images
  • Test Set B (Objects): 14,933 expressions / 4,618 sets / 750 images
  • Expression Types: Mixed (single-, multi-, no-target)
  • Use Case: Phrase-level grounding; multi-target and no-target scenarios

• RefCOCO:

  • Validation Set: 3,811 expressions / 1,500 images
  • Test Set A (People): 1,975 expressions / 750 images
  • Test Set B (Objects): 1,810 expressions / 750 images
  • Expression Style: Short (~3.6 words), allows spatial terms
  • Use Case: Single-target RES; people-vs-objects split

• RefCOCO+:

  • Validation Set: 3,805 expressions / 1,500 images
  • Test Set A (People): 1,975 expressions / 750 images
  • Test Set B (Objects): 1,798 expressions / 750 images
  • Expression Style: Short (~3.5 words), no spatial terms; appearance-based
  • Use Case: Single-target RES; appearance-based grounding

• RefCOCOg:

  • Validation Set (Google): 10,234 expressions / 4896 images
  • Validation Set (UMD): 5,000 expressions / 5,257 images
  • Test Set (UMD): 5,023 expressions / 5,096 images
  • Expression Style: Long, descriptive expressions
  • Use Case: Single-target RES; per-object split

Methodology and KPI

The evaluation metrics focus on generalized Intersection over Union (gIoU), target accuracy (T_acc), and no-target accuracy (N_acc):

• gIoU: Mean of per-image IoU values across all samples; for true positive no-target samples, IoU = 1, otherwise IoU = 0.
• T_acc: Accuracy on target samples, measuring robustness to no-target generalization.
• N_acc: Accuracy in correctly identifying no-target samples (true positive no-target predictions divided by total no-target samples).
• mask_mAP: Mean Average Precision (mAP) computed across multiple IoU thresholds (0.5 to 0.95); used to evaluate segmentation performance.

Evaluation Results: gRefCOCO

Evaluation Results: RefCOCO, RefCOCO+, RefCOCOg

Inference

Engine: Tensor(RT)
Test Hardware:

• A2
• A30
• DGX H100
• DGX A100
• Jetson AGX Xavier
• L4
• L40
• JAO 64GB
• Orin Nano 8GB
• Orin NX

The inference is run on the provided unpruned model at FP16 precision. The Jetson devices are running at Max-N configuration for maximum GPU frequency.

Technical Blogs

• Train like a ‘pro’ without being an AI expert using TAO AutoML
• Create Custom AI models using NVIDIA TAO Toolkit with Azure Machine Learning
• Developing and Deploying AI-powered Robots with NVIDIA Isaac Sim and NVIDIA TAO
• Learn endless ways to adapt and supercharge your AI workflows with TAO - Whitepaper
• Customize Action Recognition with TAO and deploy with DeepStream
• Read the two-part blog on training and optimizing 2D body pose estimation model with TAO - Part 1 | Part 2
• Learn how to train a real-time License plate detection and recognition app with TAO and DeepStream.
• Model accuracy is extremely important; learn how you can achieve state of the art accuracy for classification and object detection models using TAO.

Suggested Reading

• More information on TAO Toolkit and pre-trained models can be found at the NVIDIA Developer Zone.
• Refer to the TAO documentation.
• Read the TAO Toolkit Quick Start Guide and release notes.
• If you have any questions or feedback, see the discussions on the TAO Toolkit Developer Forums.
• Deploy your models for video analytics application using the DeepStream SDK.
• Deploy your models in Riva for ConvAI use case.

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please report security vulnerabilities or NVIDIA AI Concerns here.