PAIDF Auto-Labeling is the pseudo-labeling pipeline of the NVIDIA Physical AI Data Factory, an end-to-end service that turns raw video and image inputs into finetuning-ready training scenes through a single, configuration-driven workflow. It chains four stages—super-resolution upscaling with SeedVR2, object detection and tracking with RF-DETR paired with BoostTrack, DeepOCSORT, or ByteTrack, Vision-Language Model scene understanding, and VLM- and LLM-assisted task question generation—so that one pass yields upscaled media, per-frame detections across the eighty COCO object classes with stable track identities, dense scene metadata, timestamped event records for video inputs, and ready-to-use question sets. Much like a video summarization service, its scene-understanding stage produces structured, machine-readable event records that pinpoint precisely when and where each visible action, interaction, or anomaly occurs, each carrying a start and end time, a descriptive caption, a constrained category, and grounded references to the specific objects involved. Every stage can be enabled or disabled independently, and when super-resolution is skipped or fails, the downstream stages fall back automatically to the original input. Together these capabilities make the pipeline ideal for distilling large unlabeled corpora into supervised training signals at scale.

A defining strength of Auto-Labeling is its model flexibility. The Vision-Language and Large Language Model stages target any OpenAI-compatible endpoint, so users can point the pipeline at a locally hosted model server or at the NVIDIA API Catalog—with endpoints automatically detected when running on NVIDIA Cloud Functions—and bring the latest models to bear without code changes. Task question generation offers exactly four interchangeable approaches, ranging from a default question-driven method that combines a Vision-Language Model with a Language Model to alternatives that work over fixed time windows or directly from scene metadata, all backed by domain question banks for traffic, robotics, warehouse, and person-attribute use cases plus user-supplied custom banks. Detection is similarly tunable, from confidence and overlap thresholds to the class list and the choice of tracker, and each stage exposes a pluggable selector so new detector, tracker, Vision-Language Model, or question-generation implementations can be swapped in. All artifacts are written to a fixed output layout—contextual scene metadata, event, instance, and object records, and the generated question sets and logs.

Auto-Labeling is delivered as a containerized, configuration-driven batch pipeline rather than a long-running networked service. Users invoke it through a single command-line entry point and shape its behavior entirely with lightweight overrides against one shipped blueprint configuration, selecting stages, endpoints, GPUs, and per-domain prompts without editing files. The supported runtime is Docker, with a published NVIDIA NGC image and helper scripts that handle GPU access, host-user remapping, and API-key passthrough, and the same invocation scales from a single clip to scripted batches of many videos, with mp4-family video and common still-image formats accepted as input. Input and output roots accept both local paths and remote object-storage locations, including read-only web sources, so scenes can be staged from and uploaded back to object storage on a successful run. Results are directly consumable into downstream training and curation workflows.

Governing terms

Our containers are governed by the NVIDIA Software License Agreement and Product-Specific Terms for NVIDIA AI Products.