Retail Object Recognition
Retail Object Recognition
Model Overview
This model encodes retail objects to embedding vectors and predicts their labels based on the embedding vectors in the reference space.
In version 2.0, the Retail Object Recognition model both generates embedding vectors in a reference space and provides a classification output to categorize the objects into seen classes, which refer to the classes that have appeared in the training datasets.
Both versions 1.0 and 2.0 are adept at executing few-shot learning, having been trained through the process of metric learning. For a more comprehensive understanding of metric learning, please consult the provided references.
Model Version Summary
| Model Version | Model Architecture | Input resolution | Model Size | Training Number of Classes | Task | Training data - synthetic images | Training data - real images | Decryption Code |
|---|---|---|---|---|---|---|---|---|
| Retail Object Recognition v1.0 | ResNet101 | 224x224 | 46.7M | 100 | Recognition retail objects based on embedding vectors in reference space | 600k, 100 retail subjects | 50k, 100 retail subjects, cropped from 6 scenes | nvidia_tlt |
| Retail Object Recognition v2.0 | FAN-Base-Hybrid | 224x224 | 205.8M | 315 | This task involves a dual-output network structure, which generates an embedding vectors in reference space and a classification output for categorizing the objects into seen classes. | 80k synthetic data with improved complexity and variances compaed to training data in v1.0 | 48k, 315 retail subjects, cropped from 7 scenes (added one more scene from real training data in v1.0) | None |
Model Architecture
ResNet
Version 1.0 consists of a traunk and embedded. The trunk is a ResNet101 classification network with its fully connected layers removed. The embedder is a one-layer Perceptron with an input size of 2048 (the output dimension from the averagePool layer of the ResNet101 trunk) and an output size of 2048. Thus the embedding dimension of the Retail Object Recognition model is 2048.
FAN-Base-Hybrid
The Retail Object Recognition model (version 2.0) is composed of three primary components: a trunk, a classification head, and an embedding head. The trunk uses the FAN-Base-Hybrid (Fully Attentional Network) model architecture for feature extraction. The embedding head uses four adaptors with an output size of 448. The adaptor consists of two fully connected layers, GELU activation functions and residual connection. The classifier is derived from the adaptors along with a one-layer perceptron with an output size corresponding to the number of classes.
Training
Version 1.0
The Retail Object Recognition model (version 1.0) was trained with the Triplet Loss network algorithm. The training algorithm optimizes the network to minimize the embedding output distances (cosine similarity) between the positive images and the anchor image while maximizing the distances between the negative images and the anchor image.
The trunk and embedder use different learning rates during training. The embedder uses a smaller learning rate than the trunk for a better fine-tuning effect.
Version 2.0
For the Retail Object Recognition model (version 2.0), the training algorithm optimizes the network to minimize the embedding features and target classes for the objects. The trunk is initialized by the FAN-Base-Hybrid pretrained model and remains frozen during the training process. The embedder is trained using the Triplet Loss algorithm, an approach that enhances the network by reducing the cosine similarity-measured distances between the anchor image and positive images, while concurrently extending the distances between the anchor image and negative images. The classifier was trained through the Cross-entropy algorithm, minimizing the distance between the images and their corresponding classes. The classifier and embedder are jointly trained that can potentially enhances the precision by leveraged on the features from both branches.
Training Data
The training data of the Retail Object Recognition model was cropped from images for Retail Object Detection model training and fine-tuning data (see Retail Object Detection - TRAINING DATA). Thus it is made up of both synthetic data and real data. By mixing the synthetic and real images in the training set, the model is enhanced, allowing it to bridge the gap between simulation and reality. Consequently, the model can learn feature representations from both synthetic and real image sources.
The training data encompasses multiple angles of the retail objects, which equips the model to identify a retail object from any given angle.
Version 1.0
Specifically, the model was trained on a mixture of 0.6 million synthetic images and 50k real images. During the training phase, the triplet loss on the mixture would be optimized. And during the validation phase, the accuracy of the similarity search would be calculated. The reference data for the validation set are synthetic while the query data for the validation set are real images.
| dataset | Total # of images | Training images | Testing images |
|---|---|---|---|
| Synthetic data | 600,000 | 600,000 | - |
| Real data | 56,898 | 53,476 | 3,422 |
Version 2.0
The Retail Object Recognition model (version 2.0) trained with a new retail dataset, which total has 315 distinct categories.
Specifically, the model's training involved a combined dataset of more than 80,000 synthetic images and 48,000 real images. In the training phrase, both cross-entropy loss and triple loss are optimized with this composite dataset. In the testing phase, the probabilities for each category are computed. In parallel, a similarity search is conducted, which used synthetic data as the reference and real data as the query.
| Dataset | Total # of images | Training images | Testing images |
|---|---|---|---|
| Synthetic Data | 80,872 | 80,872 | - |
| Real Data | 129,012 | 48,140 | 58,699 |
Inference Data Ground-truth Labeling Guidelines
The real training images were cropped from Retail Object Detection datasets with ground-truth bounding-boxes and categories by human labellers. To run inference on your own datasets, you may follow the guidelines below.
Reference Data Guidelines
Reference data is the database for similarity search during the inference stage for the Retail Object Recognition model. The prediction of the inference images would be decided by the L2 distances of the extracted features. Specifically, the algorithm would select the reference object with the smallest L2 distance to the query object in the reference database by Kmeans, and the predicted class would be the corresponding class of the selected reference object.
Therefore, to achieve the highest accuracy for retail object recognition, the reference data needs to be as close to the inference data as possible, regarding the background, occlusion, object orientations, etc.
For instance, if you decide that you only want to infer the retail objects with the front face, then you can collect the front side of the retail objects only as reference data. On the other hand, if you want the Retail Object Recognition model to recognize the objects with whatever angles presented, then more orientations of the retail objects need to be collected in the reference dataset.
Generally 20-30 images/class for reference data is of the highest efficiency. However, it would be definitely better to collect more reference examples, say 100 images/class.
Below are the guidelines for the specific conditions of the images:
- All objects should take at least 70% of the frame.
- Noisy background: images with noises in the backgrounds are fine.
- Occlusion: objects occluded by distractions such as hands are fine. But at least 60% of the object should be visible.
- Truncation: objects truncated by the edge of the frame with visibility of >= 60% are fine.
- Each image should be assigned to a specific class. This model does not accommodate the case when multiple classes are classified to some “other”/”unknown” class.
Query Data Guidelines
Same as the reference data guidelines.
Notice that the Retail Object Recognition model can never correctly classify the retail object if the class is not in the reference dataset.
To get the most accurate predictions, you should avoid challenging the Retail Object Recognition model with some bad views, such as the top of a soda can (as this view can be the same across many different retail objects).
Performance
Evaluation Data
Here, we present the evaluation results of Retail Object Recognition models, including versions 1.0, 2.0, and an unreleased version 1.1. The purpose of version 1.1 is to serve as a benchmark for comparing the model architecture potentials of v1.0 and v2.0. It is important to note that all test Key Performance Indicator (KPI) data are proprietary and derived from the test KPI data of Retail Object Detection model. For more information of the test dataset, please check Retail Object Detection - TRAINING AND TEST DATA.
Methodology and KPI
The performance of Retail Object Recognition models is measured using Accuracy, which is the proportion of correct predictions (all classes) made by the model out of all predictions.
Accuracy of the Classification Head
| Model | Model Architecture | Training dataset description | Test dataset description | Accuracy |
|---|---|---|---|---|
| Retail Object Recognition - v1.0 | ResNet 101 | A mixture of 60k synthetic images and 53k real images, 100 retail subjects. Real images are obtained from 6 scenes. | 3,422 images, 100 retail subjects, 6 scenes in total | 0.8453 |
| Retail Object Recognition - v1.1 (not released) | ResNet 101 | A mixture of 80k synthetic images and 48kk real images, 315 retail subjects. Real images are obtained from 7 scenes (added one more scene based on v1.0 model training dataset) | 58,699 images, 315 retail subjects, 7 scenes in total (added one more scene based on v1.0 model test dataset) | 0.7038 |
| Retail Object Recognition - v2.0 | FAN-Base-Hybrid | Same as v1.1 | Same as v1.1 | 0.8797 |
Accuracy of the Embedding Head.
| # of test images/class | # of images/class in reference database | FAN-Base-Hybrid Accuracy |
|---|---|---|
| 180 | 1 | 0.7718 |
| 180 | 5 | 0.8432 |
| 180 | 10 | 0.8485 |
| 180 | 20 | 0.8541 |
| 180 | 30 | 0.8561 |
| 180 | 40 | 0.8557 |
| 180 | 50 | 0.8577 |
| 180 | 60 | 0.8567 |
| 180 | 70 | 0.8581 |
| 180 | 80 | 0.8591 |
| 180 | 90 | 0.8605 |
| 180 | 100 | 0.8587 |
Real-time Inference Performance
The inference is run on the provided unpruned model at FP16 precision. The model input resolution is 224x224. The inference performance is run using trtexec on Jetson AGX Orin 64GB and A10. The performance shown here is the inference only performance. The end-to-end performance with streaming video data might slightly vary depending on other bottlenecks in the hardware and software.
Version 1.0
| model | device | batch size | Latency (ms) | Images per second |
|---|---|---|---|---|
| Retail Object Recognition - v1.0 | Jetson AGX Orin 64GB | 1 | 1.59 | 627 |
| Retail Object Recognition - v1.0 | Jetson AGX Orin 64GB | 16 | 12.83 | 1247 |
| Retail Object Recognition - v1.0 | Jetson AGX Orin 64GB | 32 | 23.61 | 1356 |
| Retail Object Recognition - v1.0 | Tesla A10 | 1 | 0.98 | 1018 |
| Retail Object Recognition - v1.0 | Tesla A10 | 16 | 5.95 | 2690 |
| Retail Object Recognition - v1.0 | Tesla A10 | 64 | 20.61 | 3106 |
Version 2.0
| model | device | batch size | Latency (ms) | Images per second |
|---|---|---|---|---|
| Retail Object Recognition - v2.0 | Jetson Orin Nano | 4 | 99.50 | 40.2 |
| Retail Object Recognition - v2.0 | Orin NX 16GB | 4 | 66.89 | 59.8 |
| Retail Object Recognition - v2.0 | AGX Orin 64GB | 8 | 49.69 | 161 |
| Retail Object Recognition - v2.0 | A2 | 16 | 99.38 | 161 |
| Retail Object Recognition - v2.0 | T4 | 8 | 31.62 | 253 |
| Retail Object Recognition - v2.0 | A30 | 16 | 21.80 | 734 |
| Retail Object Recognition - v2.0 | L4 | 4 | 6.63 | 603 |
| Retail Object Recognition - v2.0 | L40 | 8 | 4.85 | 1648 |
| Retail Object Recognition - v2.0 | A100 | 64 | 39.48 | 1621 |
| Retail Object Recognition - v2.0 | H100 | 64 | 23.83 | 2686 |
How to use this model
Instructions to use unpruned model with TAO
The trainable version of this model can be used for transfer learning via TAO Toolkit, while the deployable versions can be used for inference deployment via DeepStream/TensorRT.
Instructions to transfer learning the model with TAO Toolkit
Currently we only support transfer learning for Retail Object Recognition version 1.0. To proceed with this, kindly adhere to the following steps.
Download and install TAO Toolkit.
In order to use the model as pre-trained weights for transfer learning, please use the snippet below as a template for the model component of the experiment spec file to train a
Metric Learning Recognitionmodel. Below is a code snippt to config the pretrained weights of Retail Object Recognition model v1.0 to Metric Learning Recognition transfer learning in TAO. For more details, check TAO - Metric Learning Recognition Documentation
model:
backbone: resnet_101
pretrain_choice: ""
pretrained_model_path: /path/to/retail_object_recognition.pth
input_width: 224
input_height: 224
feat_dim: 2048
train:
optim:
...
smooth_loss: True
num_epochs: 100
checkpoint_interval: 5
resume_training_checkpoint_path: null
batch_size: 16
val_batch_size: 16
report_accuracy_per_class: True
dataset:
train_dataset: /path/to/training/dataset
val_dataset:
reference: /path/to/reference/dataset
query: /path/to/query/dataset
- To start the training of
Metric Learning Recognitionmodel, use following command:
tao model ml_recog train -e=<train config file>
- For more information on experiment spec file and training with TAO Toolkit, please refer to the notebook example at TAO Toolkit - Jupyter notebooks - Retail Object Recognition.
Instructions to deploy Retail Object Recognition Model (version 1.0) with TAO Toolkit
The TAO Toolkit also provides support for deploying Retail Object Recognition model v1.0 through TAO-Deploy. The ONNX model can be converted into a TensorRT engine first and then used for inference with a reference database. To proceed with this, please follow the steps below after downloading and installing the TAO Toolkit as instructed in the last section.
- Convert the deployable (onnx file) Retail Object Recognition model v1.0 to TensorRT engine. Below is the code snippt for this conversion.
gen_trt_engine:
gpu_id: 0
onnx_file: /path/to/deployable/retail/object/recognition/v1.0
trt_engine: /path/to/converted/trt/engine/file
tensorrt:
data_type: fp32
workspace_size: 1024
min_batch_size: 1
opt_batch_size: 10
max_batch_size: 10
To launch the TensorRT conversion, use
tao deploy ml_recog gen_trt_engine -e=<trt conversion config file>
- Run Inference with the converted TensorRT engine file. Below is a code snippt to config the pretrained weights of Retail Object Recognition model v1.0 to
Metric Learning Recognitiontransfer learning in TAO.
inference:
trt_engine: /path/to/converted/trt/engine/file
input_path: /path/to/test/images
inference_input_type: image_folder
topk: 5
...
dataset:
train_dataset: ""
val_dataset:
reference: /path/to/reference/dataset
query: ""
...
To conduct the inference, use following command:
tao deploy ml_recog inference -e=<inference config file>
- For more information on experiment spec files and running inference with TAO Toolkit, please refer to the notebook example at TAO Toolkit - Jupyter notebooks - Retail Object Recognition.
Limitations
Very Small Objects
NVIDIA Retail Object Recognition models are trained to classify objects larger than 10x10 pixels. Therefore it may generate poor results when classifying objects that are smaller than 10x10 pixels.
Occluded Objects
When objects are occluded or truncated such that less than 40% of the object is visible, they may not be correctly classified by the Retail Object Detection model. Partial occlusion by hand is acceptable as the model was trained with examples having random occlusions.
Monochrome or Infrared Camera Images
The Retail Object Recognition models are trained on RGB images. Therefore, images captured in a monochrome image or IR camera image may not provide good detection results.
Warped and Blurry Images
The Retail Object Recognition models are not trained on fish-eye lense cameras or moving cameras. Therefore, the models may not perform well for warped images and images that have motion-induced or other blur.
Model versions
- Deployable_v1.0: The encrypted onnx file and encrypted etlt file for Retail Object Recognition model, inferencable on DeepStream and TAO Toolkit.
- Trainable_v1.0: The checkpoint for Retail Object Recognition model, trainable on TAO Toolkit.
- Deployable_v2.0: The decrypted onnx file for Retail Recognition model, inferencable on DeepStream.
References
Citations
Hoffer, Elad, and Nir Ailon. "Deep metric learning using triplet network." International workshop on similarity-based pattern recognition. Springer, Cham, 2015.
Na, Shi, Liu Xumin, and Guan Yong. "Research on k-means clustering algorithm: An improved k-means clustering algorithm." 2010 Third International Symposium on intelligent information technology and security informatics. Ieee, 2010.
Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Anima Anandkumar, Jiashi Feng and Jose M. Alvarez. "Understanding The Robustness in Vision Transformers". International Conference on Machine Learning (ICML). 2022
Technical blogs
- Learn how to train real-time License plate detection and recognition app with TAO and DeepStream.
- Model accuracy is extremely important, learn how you can achieve state of the art accuracy for classification and object detection models using TAO
- Learn how to train and deploy real-time intelligent video analytics apps and services using DeepStream SDK
Suggested reading
- More information on about TAO Toolkit and pre-trained models can be found at the NVIDIA Developer Zone
- Read the TAO getting Started guide and release notes.
- If you have any questions or feedback, please refer to the discussions on TAO Toolkit Developer Forums
- Deploy your model on the edge using DeepStream. Learn more about DeepStream SDK
License
License to use these models is covered by the Model EULA. By downloading the unpruned or pruned version of the model, you accept the terms and conditions of these licenses.
Ethical Considerations
NVIDIA Retail Object Recognition model classifies retail objects. However, no additional information such as people and other distractors in the background are inferred. Training and evaluation dataset mostly consists of North American content. An ideal training and evaluation dataset would additionally include content from other geographies. NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. Work with the model’s developer to ensure that it meets the requirements for the relevant industry and use case; that the necessary instruction and documentation are provided to understand error rates, confidence intervals, and results; and that the model is being used under the conditions and in the manner intended.