GestureNet Model Card
The model described in this card is a classification network, which aims to classify hand crop images into 5 gesture types:
- thumbs up
GestureNet is cascaded with hand detect or a bodypose network. For example, BodyPoseNet detects human body joints which are used to create hand crops and GestureNet acts as a classifier determining the gesture of the hand.
This is a classification model with a Resnet-18 backbone.
This model was trained using the GestureNet entrypoint in TAO. The training algorithm optimizes the network to minimize the categorical cross entropy loss for the classes.
GestureNet was trained on a proprietary dataset with more than 150K images. The training dataset contains hand crops from user facing camera at various heights with variations in user, backgrounds and illumination.
The inference performance of the GestureNet model was measured against 10000 proprietary images across a variety of environments, backgrounds and illumination.
Methodology and KPI
The KPI for the evaluation data are reported in the table below. Model is evaluated based on precision, recall and f1 score.
The inference uses FP16 precision. The inference performance runs with
trtexec on Jetson Nano, AGX Xavier, Xavier NX and NVIDIA T4 GPU. The Jetson devices run at Max-N configuration for maximum system performance. The end-to-end performance with streaming video data might slightly vary depending on use cases of applications.
This model needs to be used with NVIDIA Hardware and Software. For Hardware, the model can run on any NVIDIA GPU including NVIDIA Jetson devices. This model can only be used with Train Adapt Optimize (TAO) Toolkit, DeepStream 6.0 or TensorRT.
There are two flavors of the model:
trainable model is intended for training using TAO Toolkit and the user's own dataset. This can provide high fidelity models that are adapted to the use case. The Jupyter notebook available as a part of TAO container can be used to re-train.
deployable model is intended for efficient deployment on the edge using DeepStream or TensorRT.
deployable models are encrypted and will only operate with the following key:
- Model load key:
Please make sure to use this as the key for all TAO commands that require a model load key.
RGB Images of 160 X 160 X 3
Gesture category labels.
To create the entire end-to-end video analytic application, deploy this model with DeepStream.
The GestureNet model is designed to classify hand gestures from a camera facing the subject.
The GestureNet model is designed to classify hand gestures of subjects inside rooms with mosty monochromatic background behind the hand.
Dark-lighting, Monochrome or Infrared Camera Images
The GestureNet model was trained on RGB images in good lighting conditions. Therefore, images captured in dark lighting conditions or a monochrome image or IR camera image may not provide good detection results.
- trainable_v1.0 - Pre-trained model that is intended for training.
- deployabale_v1.0 - Deployment models that is intended to run on the inference pipeline.
- deployabale_v2.0 - Deployment models that is intended to run on the inference pipeline with int8 calibration. The calibration file is generated for TensorRT 7.
- deployabale_v2.0.1 - Deployment models that is intended to run on the inference pipeline with int8 calibration. The calibration file is generated for TensorRT 8.
- He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
- Get TAO Container
- Get other Purpose-built models from NGC model registry:
- Read the 2 part blog on training and optimizing 2D body pose estimation model with TAO - Part 1 | Part 2
- Read the technical tutorial on how PeopleNet model can be trained with custom data using Transfer Learning Toolkit
- More information on about TAO Toolkit and pre-trained models can be found at the NVIDIA Developer Zone
- Read the TAO getting Started guide and release notes.
- If you have any questions or feedback, please refer to the discussions on TAO Toolkit Developer Forums
- Deploy your model on the edge using DeepStream. Learn more about DeepStream SDK https://developer.nvidia.com/deepstream-sdk
License to use this model is covered by the Model EULA. By downloading the unpruned or pruned version of the model, you accept the terms and conditions of these licenses.
NVIDIA GestureNet model classifies the type of the gesture from a given crop. Training and evaluation dataset mostly consists of users from South Asia. An ideal training and evaluation dataset would additionally include users from other ethnicities.
NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. Work with the model’s developer to ensure that it meets the requirements for the relevant industry and use case; that the necessary instruction and documentation are provided to understand error rates, confidence intervals, and results; and that the model is being used under the conditions and in the manner intended.