NVIDIA TAO with Omniverse Replicator Synthetic Data via Rendered.ai's anatools SDK

The ana_replicator_tao resource contains a Jupyter Notebook and supporting configuration files to train the TAO DetectNet_v2 Object Detection model using Omniverse Replicator Synthetic Data generated on the Rendered.ai Platfrom. A user applies transfer learning to a Resnet-18 backbone with a targeted dataset to achieve an object detection model that is evaluated on Replicator based datasets for various occlusion levels. The notebook demonstrates the effectiveness of the TAO Toolkit, Rendered.ai's anatools, and Replicator to train CV models.

Model Training and Evaluation with NVIDIA TAO and Rendered.ai

Using This Resource

The notebook is best run in a Python environment for the TAO Toolkit Launcher. For the requirements, see the TAO Toolkit Quick Start Guide.

Traing and evaluation datasets are retrieved from the user's Rendered.ai account. Register for a free trial with Rendered.ai to get started. From the platform, create a new workspace with the content code REPLICATOR4TAO. This will load up a workspace for you that has the graphs used in the notebook.

Replicator Human Occlusion Workspace

Occlusion Experiment

Once the transfer learning is complete, the model is evaluated by retrieving datasets from your Rendered.ai organization. You can adjust the graphs to generate datasets with various occlusion levels by adding more shrub props.

Medium Occlusion Graph

The notebook has a function to estimate the average occlusion for any dataset from the occlusion value in the KITTI label. See the NVIDA Docs for a description of the Occlusion KITTI Parameter.

Effect of Occlusion on the TAO DetectNet_v2 Model

Finally, you can inference specific images to visualize the model's performance. Spot checking a batch of inferences can reveal biases in the model. With closed loop ML, you can use Rendered.ai to update the Replicator parameters, generate more training data and re-evaluate the performance.

TAO Inference on images generated with Omniverse Replicator

About Rendered.ai

Rendered.ai is a platform-as-a-service for synthetic data that enables data scientists to overcome the costs and challenges of acquiring and using real data for training machine learning and artificial intelligence systems.

Check us out: https://www.linkedin.com/company/rendered-ai