Introduction

We have introduced PipeTuner Collection. PipeTuner is an automatic tuning tool that efficiently explores the parameter space and finds the optimal parameters for the pipelines, which yields the highest KPI on the dataset provided by the user.

PipeTuner Container

Getting Started

System Requirements

PipeTuner requires the following components on an x86_64 system:

• OS Ubuntu 22.04
• NVIDIA driver 535.104 or 535.161
• Docker - Setup instructions (need to run without sudo privilege - Instructions)
• NVIDIA container toolkit - Setup instructions

NGC Setup

Users need to follow below steps to sign in to an NGC account and get an API key.

1. Visit NGC sign in page, Enter your email address and click Next, or Create an Account.
2. Choose your organization when prompted for Organization/Team. DeepStream users may use any organization and team; Metropolis Microservice users need to select nv-mdx/mdx-v2-0; Click Sign In.
3. Generate an API key following the instructions.
4. Log in to the NGC docker registry (nvcr.io) and enter the following credentials, where YOUR_NGC_API_KEY corresponds to the key you generated from the previous step.

$ docker login nvcr.io
Username: "$oauthtoken"
Password: "YOUR_NGC_API_KEY"

5. Metropolis Microservice users need to install NGC CLI following the instructions, and set ngc config as below. DeepStream users can skip this step.

$ ngc config set
Enter API key: "YOUR_NGC_API_KEY"
Enter org: nfgnkvuikvjm
Enter team: mdx-v2-0

Sample Data Setup

The sample data consists of a mini-synthetic dataset with eight 1-minute streams and config files for tuning. You can download the sample files pipe-tuner-sample.zip by clicking “Download” from PipeTuner Documentation and Sample Data page.

Once you download the sample file, unzip the file and run setup.sh to finish sample data for either DeepStream or Metropolis Microservices.

$ unzip pipe-tuner-sample.zip
$ cd pipe-tuner-sample/scripts

$ # DeepStream or Metropolis Microservices users should run only one of the following two commands based on their usage 
$ bash setup.sh deepstream            # DeepStream users

$ bash setup.sh metropolis            # Metropolis Microservices users

DeepStream users should see docker images like below.

$ docker images # bash setup.sh deepstream 
REPOSITORY                                              TAG                    
nvcr.io/nvidia/pipetuner                                1.0
nvcr.io/nvidia/deepstream                               7.0-triton-multiarch

Also, model files should be under the ‘models’ folder. They will be mapped into DeepStream containers during tuning.

$ ls ../models
labels.txt  resnet34_peoplenet_int8.etlt  resnet34_peoplenet_int8.txt  resnet50_market1501_aicity156.onnx

Metropolis users should see docker images like below. The ‘models’ folder is empty because default models in mdx-perception container will be used.

$ docker images # bash setup.sh metropolis
REPOSITORY                                              TAG                    
nvcr.io/nvidia/pipetuner                                1.0
nvcr.io/nfgnkvuikvjm/mdx-v2-0/mdx-perception            2.1

The final directory under pipe-tuner-sample is like:

pipe-tuner-sample
├── configs
│   ├── config_CameraMatrix
│   ├── config_GuiTool
│   ├── config_MTMC
│   ├── config_PGIE
│   ├── config_PipeTuner
│   └── config_Tracker
├── data
│   ├── SDG_1min_utils  
│   └── SDG_1min_videos
├── models
├── ngc_download
├── multi-camera-tracking (only for Metropolis Microservice)
└── scripts

Run the Container

The tuning process consists of two steps:

• Launch the tuning pipelines: PipeTuner runs the pipeline and measures the accuracy metric. Meanwhile it uses one or multiple optimizers to optimize the parameters in the search range against the accuracy metric;
• Retrieve and visualize the tuning results: Find the highest accuracy and the corresponding optimal parameters for deployment.

Launch the Tuning Pipelines

To run the sample pipelines, enter pipe-tuner-sample/scripts

$ cd pipe-tuner-sample/scripts

launch.sh takes in a PipeTuner config file, automatically launches the containers and starts the tuning process. Usage:

$ bash launch.sh [deepstream image name/id] [config_pipetuner.yml]

Retrieve and Visualize Tuning Results

PipeTuner provides the following features to get the tuned parameters and results.

• Plot accuracy convergence graph
• Retrieve the optimal checkpoint All below commands are executed under pipe-tuner-sample/scripts Usage: After launching tuning for some iterations, run:

$ bash result_analysis.sh [output folder] [metric]

Here output folder is the output directory created in the previous step: pipe-tuner-sample/output/PipeTuner configname.yml_output, and metric should be the same as evaluation metric defined in PipeTuner config among MOTA, IDF1 and HOTA.

License

License

NOTE: By pulling, downloading, or using PipeTuner, you accept the terms and conditions of the EULA licenses listed above.

For DeepStream SDK and Metropolis Microservices, please refer to their own licenses.

3rd Party Notice

PipeTuner container uses third-party libraries that are distributed under licenses other than PipeTuner container's own license as below.

Ethical AI

NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. Work with the model’s developer to ensure that it meets the requirements for the relevant industry and use case; that the necessary instruction and documentation are provided to understand error rates, confidence intervals, and results; and that the model is being used under the conditions and in the manner intended.