## What is Train Adapt Optimize (TAO) Toolkit? <a class="anchor" name="introduction"></a>

[Train Adapt Optimize (TAO) Toolkit](https://developer.nvidia.com/tao-toolkit) is a python based AI toolkit for taking purpose-built pre-trained AI models and customizing them with your own data. TAO adapts popular network architectures and backbones to your data, allowing you to train, fine tune, prune and export highly optimized and accurate AI models for edge deployment.

The pre-trained models accelerate the AI training process and reduce costs associated with large scale data collection, labeling, and training models from scratch. Transfer learning with pre-trained models can be used for AI applications in smart cities, retail, healthcare, industrial inspection and more.

Build end-to-end services and solutions for transforming pixels and sensor data to actionable insights using TAO, [DeepStream SDK](https://developer.nvidia.com/deepstream-sdk) and [TensorRT](https://developer.nvidia.com/tensorrt). TAO can train models for common vision AI tasks such as object detection, classification, instance segmentation as well as other complex tasks such as pose estimation, facial landmark, gaze estimation, heart rate estimation and others.

### Purpose-built Pre-Trained Models <a class="anchor" name="purposebuilt_pretrained_models"></a>

Purpose-built pre-trained models offer highly accurate AI for a variety of vision AI tasks. Developers, system builders and software partners building intelligent vision AI apps and services, can bring their own custom data and train with and fine-tune pre-trained models instead of going through the hassle of large data collection and training from scratch.

<img align="center" src="https://developer.nvidia.com/sites/default/files/akamai/NGC_Images/models/peoplenet/output_11ft45deg_000070.jpg" width="500"> 
**PeopleNet** 

<img align="center" src="https://developer.nvidia.com/sites/default/files/akamai/TLT/bodyposenet/nv_pose2d_1.png" width="500"> 
**2D Body Pose Estimation** 

<img style="center" src="https://developer.nvidia.com/sites/default/files/akamai/TLT/fpe_sample_keypoints.png" width="500"> 
**Facial Landmark Estimation** 

The purpose-built models are available on NGC. Under each model cards, there is a pruned version that can be deployed as is or an unpruned version which can be used with TAO to fine tune with your own dataset.

| **Model Name** | **Network Architecture** | **Number of classes** | **Accuracy** | **Use Case** |
|---|---|:-:|:-:|---|
| [TrafficCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:trafficcamnet)| DetectNet_v2-ResNet18 | 4 | 83.5% mAP | Detect and track cars |
| [PeopleNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplenet)| DetectNet_v2-ResNet18 | 3 | 80% mAP | People counting, heatmap generation, social distancing|
| [PeopleNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplenet)| DetectNet_v2-ResNet34 | 3 | 84% mAP | People counting, heatmap generation, social distancing|
| [DashCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:dashcamnet)| DetectNet_v2-ResNet18 | 4 | 80% mAP | Identify objects from a moving object |
| [FaceDetectIR](https://ngc.nvidia.com/catalog/models/nvidia:tao:facedetectir) | DetectNet_v2-ResNet18 | 1 | 96% mAP | Detect face in a dark environment with IR camera |
| [VehicleMakeNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:vehiclemakenet) | ResNet18 | 20 | 91% mAP | Classifying car models |
| [VehicleTypeNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:vehicletypenet) | ResNet18 | 6 | 96% mAP | Classifying type of cars as coupe, sedan, truck, etc |
| [PeopleSegNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplesegnet) | MaskRCNN-ResNet50 | 1 | 85% mAP | Creates segmentation masks around people, provides pixel |
|[PeopleSemSegNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplesemsegnet) | UNET | 1 | 92% MIOU | Creates semantic segmentation masks around people. Filters person from the background |
|[License Plate Detection](https://ngc.nvidia.com/catalog/models/nvidia:tao:lpdnet) | DetectNet_v2-ResNet18 | 1 | 98% mAP | Detecting and localizing License plates on vehicles |
|[License Plate Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tao:lprnet) | Tuned ResNet18 | 36(US) / 68(CH) | 97%(US)/99%(CH) | Recognize License plates numbers|
|[Gaze Estimation](https://ngc.nvidia.com/catalog/models/nvidia:tao:gazenet) | Four branch AlexNet based model | N/A | 6.5 RMSE | Detects person's eye gaze|
|[Facial Landmark](https://ngc.nvidia.com/catalog/models/nvidia:tao:fpenet) | Recombinator networks | N/A | 6.1 pixel error | Estimates key points on person's face|
|[Heart Rate Estimation](https://ngc.nvidia.com/catalog/models/nvidia:tao:heartratenet) | Two branch model with attention | N/A | 0.7 BPM | Estimates person's heartrate from RGB video|
|[Gesture Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tao:gesturenet) | ResNet18 | 6 | 0.85 F1 score | Recognize hand gestures|
|[Emotion Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tao:emotionnet) | 5 Fully Connected Layers | 6 | 0.91 F1 score | Recognize facial Emotion |
|[FaceDetect](https://ngc.nvidia.com/catalog/models/nvidia:tao:facenet) | DetectNet_v2-ResNet18 | 1 | 85.3 mAP | Detect faces from RGB or grayscale image|
|[2D Body Pose Estimation](https://ngc.nvidia.com/catalog/models/nvidia:tao:bodyposenet) | Single shot bottom-up | 18 | - | Estimates key joints on person's body |
|[ActionRecognitionNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:actionrecognitionnet) | 2D RGB-only Resnet18 | 5 | 82.88 | Recognizes action of a person from a sequence of images |
|[ActionRecognitionNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:actionrecognitionnet) | 3D RGB-only Resnet18 | 5 | 85.59 | Recognizes action of a person from a sequence of images |
|[PoseClassificationNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:poseclassificationnet) | ST-GCN | 6 | 89.53 | Recognizes action of a person from a sequence of skeletons |

### Architecture specific pre-trained models <a class="anchor" name="architecture_specific_pretrained_models"></a>

In addition to purpose-built models, TAO Toolkit supports the following detection architectures:

- [YOLOV4](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_object_detection)
- [YOLOV4-Tiny](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_object_detection)
- [YOLOV3](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_object_detection)
- [FasterRCNN](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_object_detection)
- [SSD](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_object_detection)
- [DSSD](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_object_detection)
- [RetinaNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_object_detection)
- [DetectNet_v2](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_detectnet_v2)
- [EfficientDet](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_efficientdet)

These detection meta-architectures can be used with 13 backbones or feature extractors with TAO. For a complete list of all the permutations that are supported by TAO, please see the matrix below:

TAO Toolkit supports instance segmentation using **[MaskRCNN](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_instance_segmentation)** architecture.

TAO Toolkit supports semantic segmentation using **[UNET](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_semantic_segmentation)** architecture.

#### Training <a class="anchor" name="training"></a>

To get started, first choose the model architecture that you want to build, then select the appropriate model card on NGC and then choose one of the supported backbones.

![LOGO](https://github.com/vpraveen-nv/model_card_images/raw/main/api/openimage_table.jpg)

### Running TAO Toolkit <a class="anchor" name="running_transfer_learning_toolkit"></a>

1. Setup your python environment using python `virtualenv` and `virtualenvwrapper`.

2. In TAO Toolkit, we have created an abstraction above the container, you will launch all your training jobs from the launcher. No need to manually pull the appropriate container, tao-launcher will handle that. You may install the launcher using pip with the following commands.

```sh
pip3 install nvidia-tao
```

3. Download the Jupyter notebooks that you are interested in from [NGC resources](https://ngc.nvidia.com/catalog/resources/nvidia:tao:cv_samples). After installing the pre-requisite, all the training steps will be run from inside the Jupyter notebook.

|**Purpose-built Model** | **Jupyter notebook** |
|--|--|
| PeopleNet | detectnet_v2/detectnet_v2.ipynb |
| TrafficCamNet | detectnet_v2/detectnet_v2.ipynb |
| DashCamNet | detectnet_v2/detectnet_v2.ipynb |
| FaceDetectIR| detectnet_v2/detectnet_v2.ipynb |
| VehicleMakeNet | classification/classification.ipynb |
| VehicleTypeNet | classification/classification.ipynb |
| PeopleSegNet | mask_rcnn/mask_rcnn.ipynb |
| License Plate Detection | detectnet_v2/detectnet_v2.ipynb |
| License Plate Recognition | lprnet/lprnet.ipynb |
| Gaze Estimation | gazenet/gazenet.ipynb |
| Facial Landmark | fpenet/fpenet.ipynb |
| Heart Rate Estimation | heartratenet/heartratenet.ipynb |
| Gesture Recognition | gesturenet/gesturenet.ipynb |
| Emotion Recognition | emotionnet/emotionnet.ipynb |
| FaceDetect | facenet/facenet.ipynb |
| 2D Body Pose Net | bpnet/bpnet.ipynb |
| PeopleSemSegNet | unet/unet_isbi.ipynb |
| ActionRecognitionNet | action_recognition_net/actionrecognitionnet.ipynb |
| PoseClassificationNet | pose_classification_net/poseclassificationnet.ipynb |
| PointPillars | pointpillars/pointpillars.md |

|**Open model architecture**| **Jupyter notebook** |
|--|--|
|DetectNet_v2| detectnet_v2/detectnet_v2.ipynb |
|EfficientDet| efficientdet/efficientdet.ipynb |
|FasterRCNN| faster_rcnn/faster_rcnn.ipynb |
|YOLOV3 | yolo_v3/yolo_v3.ipynb |
|YOLOV4 | yolo_v4/yolo_v4.ipynb |
|YOLOV4-Tiny | yolo_v4_tiny/yolo_v4_tiny.ipynb |
|SSD | ssd/ssd.ipynb |
|DSSD | dssd/dssd.ipynb |
|RetinaNet | retinanet/retinanet.ipynb |
|MaskRCNN | mask_rcnn/mask_rcnn.ipynb |
|UNET | unet/unet_isbi.ipynb |
|classification | classification/classification.ipynb |

### Using TAO Pre-trained Models <a class="anchor" name="using_tao_pretrained_models"></a>

- Get [TAO Object Detection](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_object_detection) pre-trained models for **YOLOV4, YOLOV3, FasterRCNN, SSD, DSSD, and RetinaNet** architectures from NGC model registry

- Get [TAO DetectNet_v2 Object Detection](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_detectnet_v2) pre-trained models for **DetectNet_v2** architecture from NGC model registry

- Get [TAO EfficientDet Object Detection](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_efficientdet) pre-trained models for **DetectNet_v2** architecture from NGC model registry

- Get [TAO classification](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_classification) pre-trained models from NGC model registry

- Get [TAO Instance segmentation](https://ngc.nvidia.com/catalog/models/nvidia:tao:pretrained_instance_segmentation) pre-trained models for **MaskRCNN** architecture from NGC

- Get Purpose-built models from NGC model registry:

 - [PeopleNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplenet)
 - [TrafficCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:trafficcamnet)
 - [DashCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:dashcamnet)
 - [FaceDetectIR](https://ngc.nvidia.com/catalog/models/nvidia:tao:facedetectir)
 - [VehicleMakeNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:vehiclemakenet)
 - [VehicleTypeNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:vehicletypenet)
 - [PeopleSegNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplesegnet)
 - [PeopleSemSegNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplesemsegnet)
 - [License Plate Detection](https://ngc.nvidia.com/catalog/models/nvidia:tao:lpdnet)
 - [License Plate Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tao:lprnet)
 - [Gaze Estimation](https://ngc.nvidia.com/catalog/models/nvidia:tao:gazenet)
 - [Facial Landmark](https://ngc.nvidia.com/catalog/models/nvidia:tao:fpenet)
 - [Heart Rate Estimation](https://ngc.nvidia.com/catalog/models/nvidia:tao:heartratenet)
 - [Gesture Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tao:gesturenet)
 - [Emotion Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tao:emotionnet)
 - [FaceDetect](https://ngc.nvidia.com/catalog/models/nvidia:facenet)
 - [2D Body Pose Net](https://ngc.nvidia.com/catalog/models/nvidia:tao:bodyposenet)
 - [ActionRecognitionNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:actionrecognitionnet)
 - [PoseClassificationNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:poseclassificationnet)
 - [PointPillarNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:ponet)

### License <a class="anchor" name="license"></a>

[TAO Toolkit getting Started](https://docs.nvidia.com/tao/tao-toolkit/text/tao_quick_start_guide.html)
License for TAO containers is included within the container at `workspace/EULA.pdf`. License for the pre-trained models are available with the model files. By pulling and using the Train Adapt Optimize (TAO) Toolkit container to download models, you accept the terms and conditions of these [licenses](https://developer.nvidia.com/tao-toolkit-software-license-agreement).

### Technical blogs <a class="anchor" name="technical_blogs"></a>

- Read the 2 part blog on training and optimizing 2D body pose estimation model with TAO - [Part 1](https://developer.nvidia.com/blog/training-optimizing-2d-pose-estimation-model-with-tao-toolkit-part-1) | [Part 2](https://developer.nvidia.com/blog/training-optimizing-2d-pose-estimation-model-with-tao-toolkit-part-2)
- Learn how to train [real-time License plate detection and recognition app](https://developer.nvidia.com/blog/creating-a-real-time-license-plate-detection-and-recognition-app) with TAO and DeepStream.
- Model accuracy is extremely important, learn how you can achieve [state of the art accuracy for classification and object detection models](https://developer.nvidia.com/blog/preparing-state-of-the-art-models-for-classification-and-object-detection-with-tao-toolkit/) using TAO
- Learn how to train [Instance segmentation model using MaskRCNN with TAO](https://developer.nvidia.com/blog/training-instance-segmentation-models-using-maskrcnn-on-the-tao-toolkit/)
- Learn how to improve INT8 accuracy using [Quantization aware training(QAT) with TAO](https://developer.nvidia.com/blog/improving-int8-accuracy-using-quantization-aware-training-and-the-tao-toolkit/)
- Read the technical tutorial on how [PeopleNet model can be trained with custom data using Transfer Learning Toolkit](https://devblogs.nvidia.com/training-custom-pretrained-models-using-tlt/)
- Learn how to [train and deploy real-time intelligent video analytics apps and services using DeepStream SDK](https://devblogs.nvidia.com/building-iva-apps-using-deepstream-5.0/)

### Suggested reading <a class="anchor" name="suggested_reading"></a>

- More information on about TAO Toolkit and pre-trained models can be found at the [NVIDIA Developer Zone](https://developer.nvidia.com/tao-toolkit)
- Read the [TAO getting Started](https://docs.nvidia.com/tao/tao-toolkit/text/tao_quick_start_guide.html) guide and [release notes](https://docs.nvidia.com/tao/tao-toolkit/text/release_notes.html).
- If you have any questions or feedback, please refer to the discussions on [TAO Toolkit Developer Forums](https://forums.developer.nvidia.com/c/accelerated-computing/intelligent-video-analytics/tao-toolkit/17)
- Deploy your model on the edge using DeepStream. Learn more about [DeepStream SDK](https://developer.nvidia.com/deepstream-sdk)

### Ethical AI <a class="anchor" name="ethical_ai"></a>

NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. Work with the model’s developer to ensure that it meets the requirements for the relevant industry and use case; that the necessary instruction and documentation are provided to understand error rates, confidence intervals, and results; and that the model is being used under the conditions and in the manner intended.

tao-toolkit-tf

Docker container with workflows implemented in TensorFlow as part of the Train Adapt Optimize (TAO) Toolkit.

TAO Toolkit for Computer Vision

Description

Publisher

Latest Tag

Modified

Compressed Size

Multinode Support

Multi-Arch Support

v3.22.05-beta-api (Latest) Scan Results