CenterPose Model Card - Isaac ROS

Model Overview

CenterPose is a single-stage, keypoint-based method for category-level object pose estimation. It processes unknown object instances within a recognized category using a single RGB image. The pretrained model detects the projections of 3D keypoints, estimates a 6-DoF pose, and regresses the relative 3D bounding cuboid dimensions.

Model Architecture

This model is supported two different types of backbone network as the feature extractor, including DLA34 and FAN-Small-Hybrid. The DLA34 is a standard Convolutional Neural Network (CNN) backbone, while the FAN-Small-Hybrid is a transformer-based classification backbone.

The network architecture processes a re-scaled and padded RGB image. Using the DLA34/FAN-Small-Hybrid feature extractor combined with an upsampling module, the network outputs three distinct heads that predict the 2D bounding box, projections of 3D bounding box keypoints, and cuboid dimensions.

Training

This model uses a single-stage network to make all predictions and is trained using the CenterPose entry point since TAO 5.2 in November 2023. The training algorithm optimizes the network to minimize both the focal loss and the l1 loss for all keypoints and cuboid dimensions.

Training Data

The CenterPose model was trained on the Objectron dataset, a newly introduced benchmark for monocular RGB category-level 6-DoF object pose estimation. This dataset comprises 15k annotated video clips, totaling over 4M annotated frames. Every category is marked with a 3D bounding cuboid that indicates the object's position, orientation relative to the camera, and the cuboid's dimensions.

For training and evaluation purposes, we extracted frames by temporally downsampling the original videos to 15 fps.

For symmetric objects, such as bottles, we produced multiple ground truth labels during the training phase by rotating them N times around their symmetry axis.

Accuracy and Performance

Evaluation Data

The performance of the CenterPose model during inference was evaluated using the test samples from each category in the official dataset release. These frames, originally high-resolution images of 600x800 pixels, were resized to 512x512 pixels before being processed by the CenterPose model.

Methodology and KPI

Accuracy was determined using a 3D intersection-over-union (IoU) criterion with a threshold greater than 0.5. The 2D MPE (mean pixel projection error) metric calculates the average normalized distance between the projections of 3D bounding box keypoints from both the estimated and ground truth poses. For viewpoint estimation, we present the average precision (AP) for azimuth and elevation with thresholds of 15° and 10° degrees, respectively.

For symmetric object categories, like bottles, we rotated the estimated bounding box around the symmetry axis N times (where N = 100) and assessed the prediction in relation to each rotated instance. The results reflect the instance that either maximizes the 3D IoU or minimizes the 2D pixel projection error.

Evaluation data key performance indicators (KPIs) are provided in the table below. The evaluation of the pretrained models was based on FP32 precision.

Real-time Inference Performance

The inference performance of the provided CenterPose model is evaluated at both FP16 and FP32 precisions. The model's input resolution is 512x512 pixels. The performance assessment was conducted using trtexec on a range of devices: Orin Nano 8GB, Orin NX 16GB, Jetson AGX Orin 64GB, A2, A30, A100, H100, L4, L40, and Tesla T4. In the table, "BS" stands for "batch size."

The performance data presented pertains solely to model inference. The end-to-end performance, when integrated with streaming video data, may vary slightly due to potential bottlenecks in both hardware and software.

How to use this model

These models are designed for use with NVIDIA platforms, including Jetson and x86_64 with a dGPU. To use this model in an inference pipeline in ROS 2, please consult Isaac ROS Pose Estimation.

• Note: Please use FP32 for the FAN-small-hybrid backbone CenterPose model if the TensorRT version is lower than 8.6.

Output image

Limitations

Very Small Objects

The CenterPose model was trained to identify dominant objects in the camera view. As a result, it might not detect objects who appear very small with respect to the camera view.

Occluded Objects

If objects are occluded or truncated to the extent that less than 40% of the object remains visible, the CenterPose model may not recognize them. The model can detect partially occluded objects as long as the majority of the object remains visible. Heavily occluded objects might compromise detection accuracy.

Dark-lighting, Distortion Images, Blurry Images

The CenterPose model was trained on RGB images taken under good lighting conditions and captured by a pinhole camera. Consequently, images shot in poor lighting or those exhibiting distortion or blur might not yield optimal detection results.

Model versions

• Deployable: decrypted ONNX files, inferencable on Isaac ROS pipeline.

References

Citations

• Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Anima Anandkumar, Jiashi Feng and Jose M. Alvarez. "Understanding The Robustness in Vision Transformers". International Conference on Machine Learning (ICML). 2022.

• Fisher Yu, Dequan Wang, Evan Shelhamer and Trevor Darrell. "Deep Layer Aggregation". Conference on Computer Vision and Pattern Recognition. 2018.

• Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, Matthias Grundmann. "Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations". IEEE Conference on Computer Vision and Pattern Recognition. 2021.

License

License to use these models is covered by the NVIDIA Open Model License. By downloading the model, you accept the terms and conditions of these licenses.

Ethical Considerations

NVIDIA CenterPose model estimates the object pose. However, no additional information such as people and other distractors in the background are inferred. Training and evaluation dataset mostly consists of North American content. An ideal training and evaluation dataset would additionally include content from other geographies. NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. Work with the model’s developer to ensure that it meets the requirements for the relevant industry and use case; that the necessary instruction and documentation are provided to understand error rates, confidence intervals, and results; and that the model is being used under the conditions and in the manner intended.