## BodyPoseNet Model Card

### Model Overview <a class="anchor" name="model_overview"></a>

The BodyPoseNet models described in this card are used for multi-person human pose estimation network, which aims to predict the skeleton for every person in a given input image which consists of keypoints and the connections between them. This follows a single shot bottom-up methodology and there is no need for a person detector. Hence, the compute does not scale linearly with the number of people in the scene.
The pose / skeleton output is commonly used as input for applications like activity/gesture recognition, fall detection, posture analysis, among others.

The default model predicts 18 keypoints including *nose, neck, right_shoulder, right_elbow, right_wrist, left_shoulder, left_elbow, left_wrist, right_hip, right_knee, right_ankle, left_hip, left_knee, left_ankle, right_eye, left_eye, right_ear, left_ear*.

<figure class="image">
    <center><img src="https://developer.nvidia.com/sites/default/files/akamai/TLT/bodyposenet/nv_pose2d_1.png" width="800"></center>
    <center><figcaption>Fig 1. Example illustration of BodyPoseNet output</figcaption></center>
</figure>

### Model Architecture <a class="anchor" name="model_architecture"></a>

This is a fully convolutional model with architecture consisting of a backbone network (like VGG), an initial estimation stage which does a pixel-wise prediction of confidence maps (heatmaps) and part affinity fields followed by multistage refinement (0 to N stages) on the initial predictions.

### Training Algorithm <a class="anchor" name="training_algorithm"></a>

The training algorithm optimizes the network to minimize the loss on confidence maps (heatmaps) and part affinity fields for given image and ground truth pose labels.

#### Training Data <a class="anchor" name="training_data"></a>

The available pretrained model is trained on a subset of the Google OpenImages dataset.

### Performance <a class="anchor" name="performance"></a>

#### Evaluation Dataset

The inference performance of BodyPoseNet v1.0 model was measured against COCO validation dataset.

#### Methodology and KPI <a class="anchor" name="kpi_accuracy"></a>

The KPI for the evaluation data are reported in the table below.
| Metric | IoU | Area | Score
| ----  | ---- | ---- | ---- |
| AP | 0.50:0.95 | all | 56.2 |
| AP | 0.5  | all | 79.3 |
| AP | 0.50:0.95 | medium | 57.2 |
| AP | 0.50:0.95 | large | 54.9 |

#### Real-time Inference Performance <a class="anchor" name="realtime_inference_performance"></a>

The inference performance is measured for INT8 precision and for a input dimension of 288x384. The inference performance runs with [`trtexec`](https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec) on Jetson Nano, AGX Xavier, Xavier NX and NVIDIA T4 GPU. The Jetson devices run at Max-N configuration for maximum system performance. The end-to-end performance with streaming video data might slightly vary depending on use cases of applications.

| Device | Precision | Batch_size | FPS | Latency |
| ----  | ----    | ---- | --- | --- |
| Nano | INT8 | 8 | 5 | 200.0ms |
| NX | INT8 | 8 | 93 | 10.71ms |
| Xavier | INT8 | 8 | 160 | 6.25ms |
| T4 | INT8 | 8 | 555 | 1.80ms |

### How to use this model <a class="anchor" name="how_to_use_this_model"></a>

The models in this page can only be used with [Train Adapt Optimize (TAO) Toolkit](https://developer.nvidia.com/tao-toolkit). TAO provides a simple command line interface to train a deep learning model for body pose estimation.

Primary use case for this model is to detect human poses in a given RGB image. BodyPoseNet is commonly used for activity/gesture recognition, fall detection, posture analysis etc.

1. Install the NGC CLI from ngc.nvidia.com

2. Configure the NGC CLI using the following command

```sh
ngc config set
```

3. To view all the models that are supported in TAO:

```sh
ngc registry model list nvidia/tao/bodyposenet:*
```

4. To download the model:

```sh
ngc registry model download-version nvidia/tao/bodyposenet:<template> --dest <path>
```

#### Input

Network accepts H X W x 3 input. The images are pre-processed to handle normalization, resizing while maintaining the aspect ratio etc.

#### Output

Network outputs two tensors: confidence maps (H1' x W1' x C) and part affinity fields (H2' x W2' x P). After NMS and bipartite graph matching, we obtain final results with M x N X 3

where
- N is the number of keypoints.
- M is the number of humans detected in the image.
- C is the number of confidence map channels - corresponds to number of keypoints + background
- P is the number of part affinity field channels - corresponds to the (2 x number of edges used in the skeleton)
- H1', W1' are the height and width of the output confidence maps respectively
- H2', W2' are the height and width of the output part affinity fields respectively

### Limitations

#### Crowded scenes

BodyPoseNet model does not give good results for very crowded scenes, especially if detecting the pose for small-scale people in the image.

#### Occlusions

The network may have difficulty estimating poses of people who are occluded by other objects or persons.

#### Others

The network may have difficulty estimating poses of people when there exists no distinction with the background (for example, estimation failure may occur for a person wearing a black sweater against a dark background).

### Model versions:

- **trainable_v1.0** - this pretrained model is intended to be used for finetuning on custom datasets using TAO.
- **deployabale_v1.0** - this deployable model is intended to run on the inference pipeline. There are INT8 calibration files provided for three resolutions including 224x320, 288x384 and 320x448. These calibration files are generated for TensorRT 7.
- **deployabale_v1.0.1** - this deployable model is intended to run on the inference pipeline. There are INT8 calibration files provided for three resolutions including 224x320, 288x384 and 320x448. These calibration files are generated for TensorRT 8.

The `trainable` and `deployable` models are encrypted and will only operate with the following key:

- Model load key: `nvidia_tlt`

Please make sure to use this as the key for all TAO commands that require a model load key.

### Reference <a class="anchor" name="citations"></a>

- Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, Yaser Sheikh (2017). Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

### Using TAO Pre-trained Models <a class="anchor" name="using_tao_pretrained_models"></a>

- Get [TAO Container](https://ngc.nvidia.com/catalog/containers/nvidia:tao:tao-toolkit)
- Get other Purpose-built models from NGC model registry:
  - [FaceNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:facenet)
  - [FPENet](https://ngc.nvidia.com/catalog/models/nvidia:tao:fpenet)
  - [GazeNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:gazenet)
  - [EmotionNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:emotionnet)
  - [GestureNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:gesturenet)
  - [HeartRateNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:heartratenet)
  - [BodyPoseNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:bodyposenet)
  - [PeopleNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplenet)
  - [TrafficCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:trafficcamnet)
  - [DashCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:dashcamnet)
  - [VehicleMakeNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:vehiclemakenet)
  - [VehicleTypeNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:vehicletypenet)

### Technical blogs <a class="anchor" name="technical_blogs"></a>

- Read the 2 part blog on training and optimizing 2D body pose estimation model with TAO - [Part 1](https://developer.nvidia.com/blog/training-optimizing-2d-pose-estimation-model-with-tao-toolkit-part-1)  |  [Part 2](https://developer.nvidia.com/blog/training-optimizing-2d-pose-estimation-model-with-tao-toolkit-part-2)
- Read the technical tutorial on how [PeopleNet model can be trained with custom data using Transfer Learning Toolkit](https://devblogs.nvidia.com/training-custom-pretrained-models-using-tlt/)

### Suggested reading <a class="anchor" name="suggested_reading"></a>

- More information on about TAO Toolkit and pre-trained models can be found at the [NVIDIA Developer Zone](https://developer.nvidia.com/tao-toolkit)
- Read the [TAO getting started guide](https://docs.nvidia.com/tao/tao-toolkit/) guide and [release notes](https://docs.nvidia.com/tao/tao-toolkit/text/release_notes.html).
- If you have any questions or feedback, please refer to the discussions on [TAO Toolkit Developer Forums](https://forums.developer.nvidia.com/c/accelerated-computing/intelligent-video-analytics/tao-toolkit/17)

### License <a class="anchor" name="license"></a>

This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, please visit this [link](http://creativecommons.org/licenses/by/4.0/),  or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

### Ethical Considerations <a class="anchor" name="ethical_considerations"></a>

NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. Work with the model’s developer to ensure that it meets the requirements for the relevant industry and use case; that the necessary instruction and documentation are provided to understand error rates, confidence intervals, and results; and that the model is being used under the conditions and in the manner intended.

BodyPoseNet

Description

Publisher

Use Case

Framework

Latest Version

Modified

Size