BodyPose3DNet

Model Overview

The BodyPose3DNet models described in this card are used for 3D human pose estimation network, which aims to predict the skeleton for every person in a given input image which consists of keypoints and the connections between them. 3D body pose Tracking has many practical applications such as action understanding, surveillance, human-robot interaction, motion capture and CGI, augmented and virtual reality, assisted living, advanced driver assistance systems (ADAS) and sport analysis, AI-powered sports coaches, Workplace activity monitoring, Crowd counting and tracking, allowing for character animation that doesn’t rely on markers or specialized suits.

Given an RGB image, we want to track the 2D and 3D poses of the body with 34 joints of the body: pelvis, left_hip, right_hip, torso, left_knee, right_knee, neck, left_ankle, right_ankle, left_big_toe, right_big_toe, left_small_toe, right_small_toe, left_heel, right_heel, nose, left_eye, right_eye, left_ear, right_ear, left_shoulder, right_shoulder, left_elbow, right_elbow, left_wrist, right_wrist, left_pinky_knuckle, right_pinky_knuckle, left_middle_tip, right_middle_tip, left_index_knuckle, right_index_knuckle, left_thumb_tip, right_thumb_tip. The picture demonstrates the predicted skeleton, constructed from the above 34 keypoints, overlaid on the original video frame.

\n\n

\n Fig 1. Example illustration of BodyPose3DNet output\n\n

License/Terms of Use

License to use these models is covered by the NVIDIA Open Model License. By downloading the model, you accept the terms and conditions of these licenses.

Model Architecture

Architecture Type: Deep Convolutional Neural Network.

Network Architecture: HRNet

\n\n\n\n\n

How to use this model

The models in this page can only be used with Train Adapt Optimize (TAO) Toolkit. TAO provides a simple command line interface to train a deep learning model for 3D body pose estimation.

Primary use case for this model is to detect human poses in a given RGB image. BodyPose3DNet is commonly used for activity/gesture recognition, fall detection, posture analysis etc.

\n
Install the NGC CLI from ngc.nvidia.com
\n
\n
Configure the NGC CLI using the following command
\n

ngc config set\n

To view all the models that are supported in TAO:

ngc registry model list nvidia/tao/bodypose3dnet:*\n

To download the model:

--dest \n\">

ngc registry model download-version nvidia/tao/bodypose3dnet:<template> --dest <path>\n

Input

Input tensor 0: \n    name:  input0\n    elem_type: float32\n    shape: -1 x 3 x 256 x 192\n\nInput tensor 1: \n    name:  k_inv\n    elem_type: float32\n    shape: -1 x 3 x 3\n\nInput tensor 2: \n    name:  t_form_inv\n    elem_type: float32\n    shape: -1 x 3 x 3\n\nInput tensor 3: \n    name:  scale_normalized_mean_limb_lengths\n    elem_type: float32\n    shape: -1 x 36\n\nInput tensor 4: \n    name:  mean_limb_lengths\n    elem_type: float32\n    shape: -1 x 36\n

Output

Network outputs four tensors:

Output tensor 0: \n    name: pose2d\n    elem_type: float32\n    shape: -1 x 34 x 3 [x, y, c]\n\nOutput tensor 1: \n    name: pose2d_org_img\n    elem_type: float32\n    shape: -1 x 34 x 3 [x, y, c]\n\nOutput tensor 2: \n    name: pose25d\n    elem_type: float32\n    shape: -1 x 34 x 4 [x, y, depth, c]\n\nOutput tensor 3: \n    name: pose3d\n    elem_type: float32\n    shape: -1 x 34 x 3 [x, y, z]\n

Limitations

It only supports one person in the scene.
Full body of the person needs to be visible.

Model versions

deployable_accuracy_v1.0: A deployable model with better accuracy but more computations.
deployable_performance_v1.0: A deployable model which is more optimized for inference performance.

Reference

Using TAO Pre-trained Models

Get TAO Container
Get other Purpose-built models from NGC model registry:\n
- FaceNet
- FPENet
- GazeNet
- EmotionNet
- GestureNet
- HeartRateNet
- BodyPoseNet
- BodyPose3DNet
- PeopleNet
- TrafficCamNet
- DashCamNet
- VehicleMakeNet
- VehicleTypeNet
\n

Technical blogs

Read the 2 part blog on training and optimizing 2D body pose estimation model with TAO - Part 1 | Part 2
Read the technical tutorial on how PeopleNet model can be trained with custom data using Transfer Learning Toolkit

License

This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, please visit this link, or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

Ethical Considerations

NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. Work with the model’s developer to ensure that it meets the requirements for the relevant industry and use case; that the necessary instruction and documentation are provided to understand error rates, confidence intervals, and results; and that the model is being used under the conditions and in the manner intended.

","resourceVersions",[192,193,194,195,196],"resourceVersionsTotal","hasMoreVersions","versionMetadata",{"_166":21,"_167":168,"_48":169,"_170":-5,"_171":172,"_173":174,"_175":80,"_176":16,"_177":86,"_178":-5,"_179":30,"_180":-5,"_181":30,"_182":-5,"_183":30,"_184":-5,"_185":30,"_186":-5,"_187":30,"_188":-5,"_189":30,"_190":191},"versionNotFound","isLoggedIn","latestVersionPath","/orgs/nvidia/tao/models/bodypose3dnet/-","version","imageReference","cliDownloadCommand","ngc registry model download-version nvidia/tao/bodypose3dnet:deployable_accuracy_onnx_1.0","helmFetchCommand","viewAllVersionsPath","/orgs/nvidia/tao/models/bodypose3dnet/deployable_accuracy_onnx_1.0/version-history","accessCheckResult",{"_165":86},"hasRelatedCollections","context",{"_161":162,"_163":164},1788691626758,[],[],[154,155,156,157,158,159,160],{"_44":59,"_46":70,"_48":61,"_50":62,"_52":71,"_54":55},{"_44":59,"_46":68,"_48":61,"_50":62,"_52":69,"_54":55},{"_44":59,"_46":66,"_48":61,"_50":62,"_52":67,"_54":55},{"_44":59,"_46":64,"_48":102,"_50":-5,"_52":-5,"_54":55},{"_44":59,"_46":60,"_48":61,"_50":62,"_52":63,"_54":55},{"_44":56,"_46":57,"_48":49,"_50":51,"_52":58,"_54":55},{"_44":45,"_46":47,"_48":49,"_50":51,"_52":53,"_54":55},"orgName","nvidia","teamName","tao","allowed","versionId","description","","UPLOAD_COMPLETE","releaseType","createdDate",["D",1726793610176],"updatedDate",["D",1740840131642],"totalSizeInBytes","totalFileCount","isSigned","malwareScanStatus","isScanned","releaseNotes","hasReleaseNotes","setup","hasSetup","quickStartGuide","hasQuickStartGuide","advanced","hasAdvanced","performance","hasPerformance","policies",[],{"_76":204,"_78":205,"_170":-5,"_177":86,"_179":30,"_190":203,"_181":30},{"_76":202,"_78":202,"_170":-5,"_177":86,"_179":30,"_190":203,"_181":30},{"_76":21,"_78":21,"_170":-5,"_177":86,"_179":30,"_190":201,"_181":30},{"_76":199,"_78":199,"_170":-5,"_177":30,"_179":30,"_190":200,"_181":30},{"_76":197,"_78":197,"_170":-5,"_177":30,"_179":30,"_190":198,"_181":30},"deployable_accuracy_v1.0",[],"deployable_performance_v1.0",[],[],"deployable_performance_onnx_v1.0",[],"-","Latest (deployable_performance_onnx_v1.0)","nvidia/tao/bodypose3dnet","key","nvidia/tao/models/bodypose3dnet","resourceType","models","resourceId","bodypose3dnet","shortDescription","3D human pose estimation network to predict 34 keypoints in 3D of a person in an image.","# BodyPose3DNet\n\n## Model Overview \n\nThe BodyPose3DNet models described in this card are used for 3D human pose estimation network, which aims to predict the skeleton for every person in a given input image which consists of keypoints and the connections between them. 3D body pose Tracking has many practical applications such as action understanding, surveillance, human-robot interaction, motion capture and CGI, augmented and virtual reality, assisted living, advanced driver assistance systems (ADAS) and sport analysis, AI-powered sports coaches, Workplace activity monitoring, Crowd counting and tracking, allowing for character animation that doesn’t rely on markers or specialized suits. \n\nGiven an RGB image, we want to track the 2D and 3D poses of the body with 34 joints of the body: *pelvis, left_hip, right_hip, torso, left_knee, right_knee, neck, left_ankle, right_ankle, left_big_toe, right_big_toe, left_small_toe, right_small_toe, left_heel, right_heel, nose, left_eye, right_eye, left_ear, right_ear, left_shoulder, right_shoulder, left_elbow, right_elbow, left_wrist, right_wrist, left_pinky_knuckle, right_pinky_knuckle, left_middle_tip, right_middle_tip, left_index_knuckle, right_index_knuckle, left_thumb_tip, right_thumb_tip*. The picture demonstrates the predicted skeleton, constructed from the above 34 keypoints, overlaid on the original video frame.\n\n

Fig 1. Example illustration of BodyPose3DNet output

\n\n### License/Terms of Use\nLicense to use these models is covered by the NVIDIA Open Model License. By downloading the model, you accept the terms and conditions of these [licenses](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).\n\n## Model Architecture \n\n**Architecture Type:** Deep Convolutional Neural Network.\n\n**Network Architecture:** [HRNet](https://arxiv.org/abs/1908.07919v2)\n\n\n\n\n\n\n\n\n\n## How to use this model \n\nThe models in this page can only be used with [Train Adapt Optimize (TAO) Toolkit](https://developer.nvidia.com/tao-toolkit). TAO provides a simple command line interface to train a deep learning model for 3D body pose estimation.\n\nPrimary use case for this model is to detect human poses in a given RGB image. BodyPose3DNet is commonly used for activity/gesture recognition, fall detection, posture analysis etc.\n\n1. Install the NGC CLI from ngc.nvidia.com\n\n2. Configure the NGC CLI using the following command\n\n ```sh\n ngc config set\n ```\n\n3. To view all the models that are supported in TAO:\n\n ```sh\n ngc registry model list nvidia/tao/bodypose3dnet:*\n ```\n\n4. To download the model:\n\n ```sh\n ngc registry model download-version nvidia/tao/bodypose3dnet: