Model Overview

Description:

The Semantic Segmentation model assigns every pixel in an image to a class label.

This model is ready for commercial use.

References:

Other TAO Pre-trained Models

• Get TAO Object Detection pre-trained models for YOLOV4, YOLOV3, FasterRCNN, SSD, DSSD, and RetinaNet architectures from NGC model registry

• Get TAO DetectNet_v2 Object Detection pre-trained models for DetectNet_v2 architecture from NGC model registry

• Get TAO EfficientDet Object Detection pre-trained models for DetectNet_v2 architecture from NGC model registry

• Get TAO classification pre-trained models from NGC model registry

• Get TAO Instance segmentation pre-trained models for MaskRCNN architecture from NGC

• Get Purpose-built models from NGC model registry:

  • PeopleNet
  • TrafficCamNet
  • DashCamNet
  • FaceDetectIR
  • VehicleMakeNet
  • VehicleTypeNet
  • PeopleSegNet
  • PeopleSemSegNet
  • License Plate Detection
  • License Plate Recognition
  • Facial Landmark
  • FaceDetect
  • 2D Body Pose Net
  • ActionRecognitionNet

Model Architecture:

Architecture Type: UNet
Network Architecture:

• Resnet10
• Resnet18
• Resnet34
• Resnet50
• Resnet101
• Vgg16
• Vgg19

Input:

Input Type(s): Image
Input Format(s): Red, Green, Blue (RGB)
Input Parameters: 3D
Other Properties Related to Input: Minimum Resolution: B X 3 X 224 X 224; Maximum Resolution: B X 3 X 518 X 518; No minimum bit depth, alpha, or gamma

Output:

Output Type(s): Label(s), Semantic Segmentation Mask
Output Format: Label: Text String(s); Semantic Segmentation Mask: 2D
Other Properties Related to Output: Category Label(s): (objects detected), Semantic Segmentation Mask

Software Integration:

Runtime Engine(s):

• TAO - 5.2
  
• DeepStream 6.1 or later
  

Supported Hardware Architecture(s):

• Ampere
  
• Jetson
  
• Hopper
  
• Lovelace
  
• Pascal
  
• Turing
  
• Volta
  

Supported Operating System(s):

• Linux
  
• Linux 4 Tegra
  

Model Version(s):

• Resnet10
• Resnet18
• Resnet34
• Resnet50
• Resnet101
• Vgg16
• Vgg19

Training & Evaluation:

Training Dataset:

Link: https://github.com/openimages/dataset/blob/main/READMEV3.md

Data Collection Method by dataset:

• Unknown
  

Labeling Method by dataset:

• Unknown
  

Properties:
Roughly 400,000 images and 7,000 validation images across thousands of classes as defined by Google OpenImages Version Three (3) dataset. Most of the human verifications have been done with in-house annotators at Google. A smaller part has been done with crowd-sourced verification from Image Labeler: Crowdsource app, g.co/imagelabeler.

Evaluation Dataset:

Link: https://github.com/openimages/dataset/blob/main/READMEV3.md

Data Collection Method by dataset:

• Unknown
  

Labeling Method by dataset:

• Unknown
  

Properties:
15,000 test images from Google OpenImages Version Three (3) dataset.

Inference:

Engine: Tensor(RT)
Test Hardware:

• Jetson AGX Xavier
• Xavier NX
• Orin
• Orin NX
• NVIDIA T4
• Ampere GPU
• A2
• A30
• L4
• T4
• DGX H100
• DGX A100
• DGX H100
• L40
• JAO 64GB
• Orin NX16GB
• Orin Nano 8GB

Training Semantic Segmentation Models Using TAO

The semantic segmentation apps in TAO expect mask data as encoded images with every pixel assigned to the class label. TAO provides a simple command line interface to train a deep learning model for semantic segmentation.

The models in this model area are only compatible with TAO Toolkit. For more information about using this model with TAO, please view the instructions here to install the TAO Launcher CLI and use with the semantic segmentation trainer in TAO.

Before running the container, use docker pull to ensure an up-to-date image is installed. Once the pull is complete, you can run the container image.

1. Install the NGC CLI from ngc.nvidia.com

2. Configure the NGC CLI using the following command

ngc config set

3. To view all the backbones that are supported by Instance segmentation architecture in TAO:

   ngc registry model list nvidia/tao/pretrained_semantic_segmentation:*
   

4. Download the model:

   ngc registry model download-version nvidia/tao/pretrained_semantic_segmentation:<template> --dest <path>
   

Instructions to run the sample notebook

1. Get the NGC API key from the SETUP tab on the left. Please store this key for future use. Detailed instructions can be found here

2. Configure the NGC command line interface using the command mentioned below and follow the prompts.

   ngc config set
   

3. Download the sample notebooks from NGC using the command below

   ngc registry resource download-version "nvidia/tao_cv_samples:v1.0.2"
   

4. Invoke the jupyter notebook using the following command

   jupyter notebook --ip 0.0.0.0 --port 8888 --allow-root
   

5. Open an internet browser and type in the following URL to start running the notebooks when running on a local machine.

   http://0.0.0.0:8888
   

   If you wish to run view the notebook from a remote client, please modify the URL as follows:

   http://a.b.c.d:8888
   

   Where, the a.b.c.d is the IP address of the machine running the container.

Technical blogs

• Read the 2 part blog on training and optimizing 2D body pose estimation model with TAO - Part 1 | Part 2
• Learn how to train real-time License plate detection and recognition app with TAO and DeepStream.
• Model accuracy is extremely important, learn how you can achieve state of the art accuracy for classification and object detection models using TAO
• Learn how to train Instance segmentation model using MaskRCNN with TAO
• Learn how to improve INT8 accuracy using Quantization aware training(QAT) with TAO
• Read the technical tutorial on how PeopleNet model can be trained with custom data using TAO Toolkit
• Learn how to train and deploy real-time intelligent video analytics apps and services using DeepStream SDK

Suggested reading

• More information on about TAO Toolkit and pre-trained models can be found at the NVIDIA Developer Zone
• TAO documentation
• Read the TAO getting Started guide and release notes.
• If you have any questions or feedback, please refer to the discussions on TAO Toolkit Developer Forums
• Deploy your models for video analytics application using DeepStream. Learn more about DeepStream SDK

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Promise and the Explainability, Bias, Safety & Security, and Privacy Subcards.