Holoscan Sample App Data for Point Tracking in Colonoscopy

This resource contains the TracksTo4D model for 4D point tracking in colonoscopy videos, and a sample surgical video.

Kasten, Y., Lu, W., & Maron, H. (2024). Fast encoder-based 3d from casual videos via point track processing. Advances in Neural Information Processing Systems, 37, 96150-96180.

Model

Description:

A real-time, unsupervised method for generating dynamic 4D point clouds and accurate camera tracking from colonoscopy videos, with 27x better camera pose and 1.6mm depth accuracy. Since it is unsupervised, it eliminates the need to obtain ground-truth 3D data in clinical settings. This model is ready for commercial use.

For an application of this model in Holoscan SDK, please see Holohub Link.

License/Terms of Use:

NVIDIA Open Model License

Deployment Geography:

Global

Use Case:

Clinical colonoscopy procedures for real-time 3D reconstruction and camera tracking.

References(s):

TracksTo4D: Kasten, Y., Lu, W., & Maron, H. (2024). Fast encoder-based 3d from casual videos via point track processing. Advances in Neural Information Processing Systems, 37, 96150-96180.

Model Architecture:

• Architecture Type: Transformer
• Network Architecture: TracksTo4D
• Number of model parameters: 12M

Input(s):

• Input Type(s): 2D Point Tracking from Colonoscopy Videos
• Input Format(s): Tensor
• Input Parameters: Two-Dimensional (2D)
• Other Properties Related to Input: The model takes 2D feature tracks from colonoscopy videos as input. Tracking points projected to the screen space.

Output(s)

• Output Type(s): 4D Point Cloud, Camera Pose
• Output Format(s): Tensor, Tensor
• Output Parameters: Three-Dimensional (3D), Three-Dimensional (3D)
• Other Properties Related to Output: Dynamic 4D point clouds and 6-DoF camera tracking.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s):

• TensorRT

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
• NVIDIA Blackwell
• NVIDIA Hopper

Preferred/Supported Operating System(s):

• Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

This AI model can be used with Holoscan for optimal performance.

Model Version(s):

v1.0

Training, Testing, and Evaluation Datasets:

Dataset Overview

Total Size: 2.7 Million Frames Total Number of Datasets: 1 Dataset partition: 100%

Given that the raw footage suffered from quality issues such as motion blur, fluid occlusion, and camera-wall collisions, we implemented a geometric curation pipeline based on Structure-from-Motion (SfM). We processed the data into 20-frame sequences, estimating camera poses and sparse 3D structure, followed by refinement via Bundle Adjustment and metric auto-calibration. Crucially, this pipeline handles the dynamic nature of the colon by isolating a subset of static points that satisfy rigid multiview constraints. Sequences were then automatically filtered for high geometric consistency, defined by low reprojection errors and a high inlier count, yielding a high-quality subset of 37,000 sequences (740,000 frames) selected from approximately 2 million processed frames.

Public Datasets

REAL-colon dataset: https://plus.figshare.com/articles/media/REAL-colon_dataset/22202866

Training Dataset:

Data Modality: Video

Video Training Data Size: Less than 10,000 Hours

Data Collection Method by dataset: Human

Properties: 29,535 sequences from colonoscopy recordings (REAL-colon dataset).

Testing Dataset:

Data Collection Method by dataset: Human

Labeling Method by dataset: Not Applicable

Properties 8,137 sequences from colonoscopy recordings (REAL-colon dataset), selected from distinct clinical procedures to better assess generalization.

Evaluation Dataset:

Benchmark Score

Camera pose:

• ATE (dm): 0.012
• RTE (cm): 0.003
• ROT (deg): 0.043

Depth error:

• L1 (cm): 0.161
• Relative error (%): 4.14
• RMSE (cm): 0.22

Data Collection Method by dataset: Human

Properties 8,137 sequences from colonoscopy recordings (REAL-colon dataset), selected from distinct clinical procedures to better assess generalization.

Inference:

Acceleration Engine: TensorRT

Test Hardware:

• A100, RTX 6000 ADA, RTX A6000

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the Bias, Explainability, Safety & Security, and Privacy Subcards.

Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

License

Refer to the license agreement for use of the sample data.