## What is Transfer Learning Toolkit? <a class="anchor" name="introduction"></a>

[Transfer Learning Toolkit](https://developer.nvidia.com/transfer-learning-toolkit) (TLT) is a python based AI toolkit for taking purpose-built pre-trained AI models and customizing them with your own data. TLT adapts popular network architectures and backbones to your data, allowing you to train, fine tune, prune and export highly optimized and accurate AI models for edge deployment.

The pre-trained models accelerate the AI training process and reduce costs associated with large scale data collection, labeling, and training models from scratch. Transfer learning with pre-trained models can be used for AI applications in smart cities, retail, healthcare, industrial inspection and more.

Build end-to-end services and solutions for transforming pixels and sensor data to actionable insights using TLT, [DeepStream SDK](https://developer.nvidia.com/deepstream-sdk) and [TensorRT](https://developer.nvidia.com/tensorrt). TLT can train models for common vision AI tasks such as object detection, classification, instance segmentation as well as other complex tasks such as pose estimation, facial landmark, gaze estimation, heart rate estimation and others.

### Purpose-built Pre-Trained Models <a class="anchor" name="purposebuilt_pretrained_models"></a>

Purpose-built pre-trained models offer highly accurate AI for a variety of vision AI tasks. Developers, system builders and software partners building intelligent vision AI apps and services, can bring their own custom data and train with and fine-tune pre-trained models instead of going through the hassle of large data collection and training from scratch.

<img align="center" src="https://developer.nvidia.com/sites/default/files/akamai/NGC_Images/models/peoplenet/output_11ft45deg_000070.jpg" width="500"> 
**PeopleNet** 

<img align="center" src="https://developer.nvidia.com/sites/default/files/akamai/TLT/bodyposenet/nv_pose2d_1.png" width="500"> 
**2D Body Pose Estimation** 

<img style="center" src="https://developer.nvidia.com/sites/default/files/akamai/TLT/fpe_sample_keypoints.png" width="500"> 
**Facial Landmark Estimation** 

The purpose-built models are available on NGC. Under each model cards, there is a pruned version that can be deployed as is or an unpruned version which can be used with TLT to fine tune with your own dataset.

| **Model Name** | **Network Architecture** | **Number of classes** | **Accuracy** | **Use Case** |
|---|---|:-:|:-:|---|
| [TrafficCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_trafficcamnet)| DetectNet_v2-ResNet18 | 4 | 83.5% mAP | Detect and track cars |
| [PeopleNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_peoplenet)| DetectNet_v2-ResNet18 | 3 | 80% mAP | People counting, heatmap generation, social distancing|
| [PeopleNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_peoplenet)| DetectNet_v2-ResNet34 | 3 | 84% mAP | People counting, heatmap generation, social distancing|
| [DashCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_dashcamnet)| DetectNet_v2-ResNet18 | 4 | 80% mAP | Identify objects from a moving object |
| [FaceDetectIR](https://ngc.nvidia.com/catalog/models/nvidia:tlt_facedetectir) | DetectNet_v2-ResNet18 | 1 | 96% mAP | Detect face in a dark environment with IR camera |
| [VehicleMakeNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_vehiclemakenet) | ResNet18 | 20 | 91% mAP | Classifying car models |
| [VehicleTypeNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_vehicletypenet) | ResNet18 | 6 | 96% mAP | Classifying type of cars as coupe, sedan, truck, etc |
| [PeopleSegNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_peoplesegnet) | MaskRCNN-ResNet50 | 1 | 85% mAP | Creates segmentation masks around people, provides pixel |
|[PeopleSemSegNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_peoplesemsegnet) | UNET | 1 | 92% MIOU | Creates semantic segmentation masks around people. Filters person from the background |
|[License Plate Detection](https://ngc.nvidia.com/catalog/models/nvidia:tlt_lpdnet) | DetectNet_v2-ResNet18 | 1 | 98% mAP | Detecting and localizing License plates on vehicles |
|[License Plate Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tlt_lprnet) | Tuned ResNet18 | 36(US) / 68(CH) | 97%(US)/99%(CH) | Recognize License plates numbers|
|[Gaze Estimation](https://ngc.nvidia.com/catalog/models/nvidia:tlt_gazenet) | Four branch AlexNet based model | N/A | 6.5 RMSE | Detects person's eye gaze|
|[Facial Landmark](https://ngc.nvidia.com/catalog/models/nvidia:tlt_fpenet) | Recombinator networks | N/A | 6.1 pixel error | Estimates key points on person's face|
|[Heart Rate Estimation](https://ngc.nvidia.com/catalog/models/nvidia:tlt_heartratenet) | Two branch model with attention | N/A | 0.7 BPM | Estimates person's heartrate from RGB video|
|[Gesture Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tlt_gesturenet) | ResNet18 | 6 | 0.85 F1 score | Recognize hand gestures|
|[Emotion Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tlt_emotionnet) | 5 Fully Connected Layers | 6 | 0.91 F1 score | Recognize facial Emotion |
|[FaceDetect](https://ngc.nvidia.com/catalog/models/nvidia:tlt_facenet) | DetectNet_v2-ResNet18 | 1 | 85.3 mAP | Detect faces from RGB or grayscale image|
|[2D Body Pose Estimation](https://ngc.nvidia.com/catalog/models/nvidia:tlt_bodyposenet) | Single shot bottom-up | 18 | - | Estimates key joints on person's body |

### Architecture specific pre-trained models <a class="anchor" name="architecture_specific_pretrained_models"></a>

In addition to purpose-built models, Transfer Learning Toolkit supports the following detection architectures:
- [YOLOV4](https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_object_detection)
- [YOLOV3](https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_object_detection)
- [FasterRCNN](https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_object_detection)
- [SSD](https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_object_detection)
- [DSSD](https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_object_detection)
- [RetinaNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_object_detection)
- [DetectNet_v2](https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_detectnet_v2)

These detection meta-architectures can be used with 13 backbones or feature extractors with TLT. For a complete list of all the permutations that are supported by TLT, please see the matrix below:

TLT3.0 supports instance segmentation using **[MaskRCNN](https://ngc.nvidia.com/catalog/models/nvidia:tlt_instance_segmentation)** architecture.

TLT3.0 supports semantic segmentation using **[UNET](https://ngc.nvidia.com/catalog/models/nvidia:tlt_semantic_segmentation)** architecture.

#### Training <a class="anchor" name="training"></a>

To get started, first choose the model architecture that you want to build, then select the appropriate model card on NGC and then choose one of the supported backbones.

![LOGO](https://developer.nvidia.com/sites/default/files/akamai/TLT/tlt_matrix2_updated.png)

### Running Transfer Learning Toolkit <a class="anchor" name="running_transfer_learning_toolkit"></a>

1. Setup your python environment using python `virtualenv` and `virtualenvwrapper`.

2. In TLT3.0, we have created an abstraction above the container, you will launch all your training jobs from the launcher. No need to manually pull the appropriate container, tlt-launcher will handle that. You may install the launcher using pip with the following commands.

```sh
pip3 install nvidia-pyindex
pip3 install nvidia-tlt
```

3. Download the Jupyter notebooks that you are interested in from [NGC resources](https://ngc.nvidia.com/catalog/resources/nvidia:tlt_cv_samples). After installing the pre-requisite, all the training steps will be run from inside the Jupyter notebook.

|**Purpose-built Model** | **Jupyter notebook** |
|--|--|
| PeopleNet | detectnet_v2/detectnet_v2.ipynb |
| TrafficCamNet | detectnet_v2/detectnet_v2.ipynb |
| DashCamNet | detectnet_v2/detectnet_v2.ipynb |
| FaceDetectIR| detectnet_v2/detectnet_v2.ipynb |
| VehicleMakeNet | classification/classification.ipynb |
| VehicleTypeNet | classification/classification.ipynb |
| PeopleSegNet | mask_rcnn/mask_rcnn.ipynb |
| License Plate Detection | detectnet_v2/detectnet_v2.ipynb |
| License Plate Recognition | lprnet/lprnet.ipynb |
| Gaze Estimation | gazenet/gazenet.ipynb |
| Facial Landmark | fpenet/fpenet.ipynb |
| Heart Rate Estimation | heartratenet/heartratenet.ipynb |
| Gesture Recognition | gesturenet/gesturenet.ipynb |
| Emotion Recognition | emotionnet/emotionnet.ipynb |
| FaceDetect | facenet/facenet.ipynb |
| 2D Body Pose Net | bpnet/bpnet.ipynb |
| PeopleSemSegNet | unet/unet_isbi.ipynb |

|**Open model architecture**| **Jupyter notebook** |
|--|--|
|DetectNet_v2| detectnet_v2/detectnet_v2.ipynb |
|FasterRCNN| faster_rcnn/faster_rcnn.ipynb |
|YOLOV3 | yolo_v3/yolo_v3.ipynb |
|YOLOV4 | yolo_v4/yolo_v4.ipynb |
| SSD | ssd/ssd.ipynb |
|DSSD | dssd/dssd.ipynb |
| RetinaNet | retinanet/retinanet.ipynb |
|MaskRCNN | mask_rcnn/mask_rcnn.ipynb |
|UNET | unet/unet_isbi.ipynb |
|classification | classification/classification.ipynb |

### Using TLT Pre-trained Models <a class="anchor" name="using_tlt_pretrained_models"></a>

- Get [TLT Object Detection](https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_object_detection) pre-trained models for **YOLOV4, YOLOV3, FasterRCNN, SSD, DSSD, and RetinaNet** architectures from NGC model registry

- Get [TLT DetectNet_v2 Object Detection](https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_detectnet_v2) pre-trained models for **DetectNet_v2** architecture from NGC model registry

- Get [TLT classification](https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_classification) pre-trained models from NGC model registry

- Get [TLT Instance segmentation](https://ngc.nvidia.com/catalog/models/nvidia:tlt_instance_segmentation) pre-trained models for **MaskRCNN** architecture from NGC

- Get Purpose-built models from NGC model registry:

 - [PeopleNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_peoplenet)
 - [TrafficCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_trafficcamnet)
 - [DashCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_dashcamnet)
 - [FaceDetectIR](https://ngc.nvidia.com/catalog/models/nvidia:tlt_facedetectir)
 - [VehicleMakeNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_vehiclemakenet)
 - [VehicleTypeNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_vehicletypenet)
 - [PeopleSegNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_peoplesegnet)
 - [PeopleSemSegNet](https://ngc.nvidia.com/catalog/models/nvidia:tlt_peoplesemsegnet)
 - [License Plate Detection](https://ngc.nvidia.com/catalog/models/nvidia:tlt_lpdnet)
 - [License Plate Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tlt_lprnet)
 - [Gaze Estimation](https://ngc.nvidia.com/catalog/models/nvidia:tlt_gazenet)
 - [Facial Landmark](https://ngc.nvidia.com/catalog/models/nvidia:tlt_fpenet)
 - [Heart Rate Estimation](https://ngc.nvidia.com/catalog/models/nvidia:tlt_heartratenet)
 - [Gesture Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tlt_gesturenet)
 - [Emotion Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tlt_emotionnet)
 - [FaceDetect](https://ngc.nvidia.com/catalog/models/nvidia:facenet)
 - [2D Body Pose Net](https://ngc.nvidia.com/catalog/models/nvidia:tlt_bodyposenet)

### License <a class="anchor" name="license"></a>

[TLT Getting Started]([TLT getting Started](https://docs.nvidia.com/tlt/tlt-user-guide/text/tlt_quick_start_guide.html)
License for TLT containers is included within the container at `workspace/EULA.pdf`. License for the pre-trained models are available with the model files. By pulling and using the Transfer Learning Toolkit SDK (TLT) container to download models, you accept the terms and conditions of these [licenses](https://developer.nvidia.com/tlt-30-ga-eula.pdf).

### Technical blogs <a class="anchor" name="technical_blogs"></a>

- Read the 2 part blog on training and optimizing 2D body pose estimation model with TLT - [Part 1](https://developer.nvidia.com/blog/training-optimizing-2d-pose-estimation-model-with-tlt-part-1) | [Part 2](https://developer.nvidia.com/blog/training-optimizing-2d-pose-estimation-model-with-tlt-part-2)
- Learn how to train [real-time License plate detection and recognition app](https://developer.nvidia.com/blog/creating-a-real-time-license-plate-detection-and-recognition-app) with TLT and DeepStream.
- Model accuracy is extremely important, learn how you can achieve [state of the art accuracy for classification and object detection models](https://developer.nvidia.com/blog/preparing-state-of-the-art-models-for-classification-and-object-detection-with-tlt/) using TLT
- Learn how to train [Instance segmentation model using MaskRCNN with TLT](https://developer.nvidia.com/blog/training-instance-segmentation-models-using-maskrcnn-on-the-transfer-learning-toolkit/)
- Learn how to improve INT8 accuracy using [Quantization aware training(QAT) with TLT](https://developer.nvidia.com/blog/improving-int8-accuracy-using-quantization-aware-training-and-the-transfer-learning-toolkit/)
- Read the technical tutorial on how [PeopleNet model can be trained with custom data using Transfer Learning Toolkit](https://devblogs.nvidia.com/training-custom-pretrained-models-using-tlt/)
- Learn how to [train and deploy real-time intelligent video analytics apps and services using DeepStream SDK](https://devblogs.nvidia.com/building-iva-apps-using-deepstream-5.0/)

### Suggested reading <a class="anchor" name="suggested_reading"></a>

- More information on about Transfer learning toolkit and pre-trained models can be found at the [NVIDIA Developer Zone](https://developer.nvidia.com/transfer-learning-toolkit)
- Read the [TLT getting Started](https://docs.nvidia.com/tlt/tlt-user-guide/text/tlt_quick_start_guide.html) guide and [release notes](https://docs.nvidia.com/tlt/tlt-user-guide/text/release_notes.html).
- If you have any questions or feedback, please refer to the discussions on [TLT Developer Forums](https://devtalk.nvidia.com/default/board/417/transfer-learning-toolkit/)
- Deploy your model on the edge using DeepStream. Learn more about [DeepStream SDK](https://developer.nvidia.com/deepstream-sdk)

### Ethical AI <a class="anchor" name="ethical_ai"></a>

NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. Work with the model’s developer to ensure that it meets the requirements for the relevant industry and use case; that the necessary instruction and documentation are provided to understand error rates, confidence intervals, and results; and that the model is being used under the conditions and in the manner intended.

tlt-streamanalytics

NVIDIA’s Transfer Learning Toolkit is a python-based AI training toolkit that allows developers to train faster and accurate neural networks on the popular deep learning architectures. Create accurate and efficient AI models for Intelligent Video Analytics and Computer Vision without expertise in AI frameworks.

Transfer Learning Toolkit for Video Streaming Analytics

Description

Publisher

Latest Tag

Modified

Compressed Size

Multinode Support

Multi-Arch Support

v3.0-py3 (Latest) Scan Results