The model described in this card is a re-identification network, which aims to generate embeddings for identifying objects captured in different scenes. Under the hood, the current backbone of the network is ResNet50. A pre-trained ReIdentificatioNet model based on public datasets is delivered. The model is trained on 751 unique IDs from the Market-1501 dataset and 156 unique IDs from the MTMC people tracking dataset of the 2023 AI City Challenge, where 148 of the IDs are synthetic.
The model has ResNet50 architecture. It will take cropped images of the objects as input and produce embeddings as output.
The training algorithm optimizes the network to minimize the triplet loss, center loss and cross entropy loss.
The model is trained on the Market-1501 dataset with 751 annotated people and a sampled version of the MTMC people tracking dataset of the 2023 AI City Challenge with 156 annotated people. The dataset statistics are as follows:
|subset||no. total identities||no. total images||no. total cameras||no. real identities||no. real images||no. real cameras||no. synthetic identities||no. synthetic images||no. synthetic cameras|
The data format must be in the following format.
/data /market1501 /bounding_box_train 0001_c1s1_01_00.jpg 0001_c1s1_02_00.jpg 0002_c1s1_03_00.jpg 0002_c1s1_04_00.jpg 0003_c1s1_05_00.jpg 0003_c1s1_06_00.jpg ... ... ... N.png /bounding_box_test 0001_c1s1_01_00.jpg 0001_c1s1_02_00.jpg 0002_c1s1_03_00.jpg 0002_c1s1_04_00.jpg 0003_c1s1_05_00.jpg 0003_c1s1_06_00.jpg ... ... ... N.jpg /query 0001_c1s1_01_00.jpg 0001_c1s1_02_00.jpg 0002_c1s1_03_00.jpg 0002_c1s1_04_00.jpg 0003_c1s1_05_00.jpg 0003_c1s1_06_00.jpg ... ... ... N.jpg
The dataset should be divided into different directories by train, test and query folders. Each of these folders will contain image crops with the above naming scheme.
For example:, the image
0001_c1s1_01_00.jpg is the first sequence
s1 of camera
01 is the first frame in the sequence
0001_c1s1_01_00.jpg is the unique ID assigned to the object. Data after the third
_ are ignored.
As shown in the class distribution table above, the test set contains the same identities of the query set. The goal is to identify test samples of the same identities for each query.
The key performance indicators are the ranked accuracy of re-identification and the mean average precision (mAP).
Rank-K accuracy: It is method of computing accuracy where the top-K highest confidence labels are matched with a ground truth label. If the ground truth label falls in one of these top-K labels, we state that this prediction is accurate. It allows us to get an overall accuracy measurement while being lenient on the predictions if the number of classes are too high and too similar. In our case, we compute rank-1, 5 and 10 accuracies. This means in case of rank-10, for a given sample, if the top-10 highest confidence labels predicted, match the label of ground truth, this sample will be counted as a correct measurement.
Mean average precision(mAP): Precision measures how accurate predictions are, in our case the logits of ID of an object. In other words, it measures the percentage of the predictions that are correct. mAP (mean average precision) is the average of average precision (AP) where AP is computed for each class, in our case ID.
|model||feature dimension||mAP (%)||rank-1 accuracy (%)||rank-5 accuracy (%)||rank-10 accuracy (%)|
The inference performance runs with
trtexec on NVIDIA Ampere and Jetson GPUs. The end-to-end performance with image data might slightly vary depending on use cases of applications.
|Model||Device||Precision||Batch Size||Latency (ms)||Images per Second|
|ResNet50||Jetson AGX Orin||FP16||1||0.96||1043.62|
|ResNet50||Jetson AGX Orin||FP16||16||6.42||2492.26|
|ResNet50||Jetson AGX Orin||FP16||64||23.09||2771.60|
This model needs to be used with NVIDIA Hardware and Software. For Hardware, the model can run on any NVIDIA GPU including NVIDIA Jetson devices. This model can only be used with Train Adapt Optimize (TAO) Toolkit, DeepStream SDK or TensorRT.
Primary use case intended for this model is to generate embeddings for an object and then perform similarity matching.
A pre-trained model is provided:
It is intended for training and fine-tune using Train Adapt Optimize (TAO) Toolkit and the users' dataset of re-identification. High fidelity models can be trained to the new use cases. The Jupyter notebook available as a part of TAO container can be used to re-train.
The model is also intended for easy deployment to the edge using DeepStream SDK or TensorRT. DeepStream provides facility to create efficient video analytic pipelines to capture, decode and pre-process the data before running inference.
The model is encrypted and can be decrypted with the following key:
Please make sure to use this as the key for all TAO commands that require a model load key.
B X 3 X 256 X 128 (B C H W)
The feature embedding
In order to use the model as pre-trained weights for transfer learning, please use the snippet below as a template for the
model component of the experiment spec file to train a ReIdentificationNet. For more information on experiment spec file, please refer to the Train Adapt Optimize (TAO) Toolkit User Guide.
model: backbone: resnet_50 last_stride: 1 pretrain_choice: imagenet pretrained_model_path: /path/to/pretrained_resenet50.pth input_channels: 3 input_width: 128 input_height: 256 neck: bnneck feat_dim: 256 neck_feat: after metric_loss_type: triplet with_center_loss: False with_flip_feature: False label_smooth: True
To create the entire end-to-end video analytic application, deploy this model with Triton Inference Server. NVIDIA Triton Inference Server is an open-source inference serving software that helps standardize model deployment and execution and delivers fast and scalable AI in production. Triton supports direct integration of this model into the server and inference from a client.
NVIDIA ReIdnetificationNet is trained on the Market-1501 dataset with 751 unique person classes and a sampled version of the MTMC people tracking dataset of the 2023 AI City Challenge with 156 unique person classes. It is expected that the accuracy of the model on external images is not at the same level as the number reported in performance section.
In general, to get better accuracy, more labeled data are needed to fine-tune the pre-trained model through TAO Toolkit.
License to use the model is covered by the Model EULA. By downloading the unpruned or pruned version of the model, you accept the terms and conditions of these licenses.
NVIDIA ReIdentificationNet model creates embeddings for identifying objects captured in different scenes.
NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. Work with the model’s developer to ensure that it meets the requirements for the relevant industry and use case; that the necessary instruction and documentation are provided to understand error rates, confidence intervals, and results; and that the model is being used under the conditions and in the manner intended.