USD Search API

USD Search API is a collection of cloud-native microservices that enable developers, creators, and workflow specialists to efficiently search through vast collections of OpenUSD data, images, and other assets using natural language or image-based inputs.

With these production-ready microservices, developers can deploy USD Search API onto their own infrastructure. With USD Search API’s artificial intelligence (AI) features, you can quickly locate untagged and unstructured 3D data and digital assets, saving time navigating unstructured, untagged 3D data. USD Search API is capable of searching and indexing 3D asset databases, as well as navigating complex 3D scenes to perform spatial searches, without requiring manual tagging of assets.

For more information please refer to USD Search API documentation.

1.1 - GA Release

Release Highlights

🔥 GA release - USD Search API

USD Search API 1.1 is an general availability release of USD Search API featuring various updates and improvements across all components. With this release, USD Search API added a new capability - VLM-based auto-tagging and auto-captioning to enable extraction of custom metadata from assets that could be customized towards target user requirements. Further, support for a broad range of backends added through S3proxy functionality.

🔥 VLM-based auto-tagging and auto-captioning

This allows to automatically tag and caption various assets stored on the storage backend using an external Vision Language Model (VLM) service. The following types of VLM services providers could be configured with the helm chart:

• NVIDIA AI Endpoints
• Azure OpenAI
• OpenAI
• Anthropic
• Mistralai

Together with the VLM type it is possible to configure the type of metadata that is being extracted from assets. By default, the following information is extracted:

• Object type - general category or type of the object.
• Scene type - context or setting in which the object is typically found.
• Colors - colors of the object.
• Materials - an estimate of what kind of materials the object could be composed of, based on the the appearance and texture information.
• Caption - short, descriptive text about the object, focusing on its unique or distinguishing features.
• Tags - searchable terms and keywords that encapsulate the essence of the 3D object, making it easily searchable and identifiable.

🔥 Various storage backend support

By utilizing S3proxy functionality, USD Search API now supports a broad range of backends. For convenience a sub-chart that configures S3proxy is included as part of the USD Search API helm chart.

Getting Started

• Collection of scripts that simplify installation of USD Search on a kubernetes cluster.
• Helm chart installation documentation
• General information

The full list of new features and capabilities is outlined below.

Release Notes

1.1.0 GA

Release Date: February 2025

• VLM-based auto-captioning and auto-tagging
• Support for a broad range of storage backends through S3proxy functionality.
• Search by custom metadata fields extracted from USD assets.

Added

• VLM-based automatic tagging and captioning.
  • allows configuring USD Search to use a broad variety of VLM providers (e.g. NVIDIA AI Endpoints, OpenAI, Azure OpenAI, Mistralai, Anthropic)
• Support for a broad variety of backends through S3proxy.
  • for convenience the helm sub-chart for s3proxy configuration is provided.
• Filer search results by applying regex on Asset URLs
• Search by custom metadata fields extracted from USD files.
  • Added support for searching by metadata stored in the customLayerData dict.
• On-demand indexing endpoint that allows to prioritize processing of selected assets without the need to wait for the background job to catch up.
• Make sure size search is happening over bounding box dimensions that are scaled by MPU

Known Limitations

• USD Search API relies on Omniverse Kit for rendering and extracting scene prim hierarchy. Therefore the service should work with any scene that Omniverse Kit can work with.
• When using the filter_by_properties query parameter in USD Search API, different filters have to be separated by a single comma (having a comma and a space is not supported).
• At the moment, asset previews are returned together in a single response, which limits the maximum number of results to be returned to around 700. If assets are requested without previews, this number is significantly higher (around 10000).
• Pagination of search results is currently not supported.
• Single and double quotes are not supported as part of the filter_by_properties query parameter.

1.0.0 GA

Release Date: November 2024

• GA release of USD Search API, including a downloadable Helm chart for self-deployment to interface with your own data.
• Beta feature added - Asset Graph Search to enable In-Scene Search, Spatial Search, USD Property Queries, Asset Dependency Tracking, Combined Query Capabilities .
• Added Python Client Library available in GitHub for ease of development.

Added

• Added Asset Graph Search [Beta] new feature
  • In-Scene Search
    • Locate specific assets within scenes using natural language or image similarity.
  • Spatial Search
    • Execute proximity queries using coordinates or relative to specific prims.
    • Find objects within a specified bounding box or radius.
    • Results can be sorted by distance, with options for vector alignment using a transformation matrix.
  • USD Property Queries
    • Query objects in a 3D scene using USD properties, such as finding all assets with a specific semantic label.
  • Asset Dependency Tracking
    • Build scene dependency graphs from USD, material, and MDL references. Use cases include: tracking assets impacted by updates, tracking assets reusability, building manifests, and optimizing workflows and scene loading.
  • Combined Query Capabilities
    • Enable complex scenarios for enhanced scene understanding, manipulation, and generation by combining different query types.
• Added USD Search API Python Client library for ease of development - available in Github.
• Added in-scene search into sample kit extension for ease of development.
• Added Terraform module for quickstart with Amazon EKS for ease of development.
• Added indexing status endpoint that allows checking processing state for individual assets.
• Storage backend - added Amazon S3 as official support in addition to Enterprise Nucleus server.

Improved

• Improved Quick startup scripts and general installation documentation for ease of development.
• Improved background indexing speed for performance improvements.

System Requirements

• Kubernetes cluster with the following features enabled:
  • Role-based access control (RBAC) - required for creating GPU-based asset rendering jobs
  • NVIDIA k8s device plugin - required for execution of asset rendering jobs. Alternatively, NVIDIA GPU-operator (which includes NVIDIA k8s device plugin) could be used.
  • Asset rendering jobs require RTX-GPUs to be available on the cluster.
  • Metrics server - required for Horizontal Pod Autoscaling (HPA)
• Hardware
  • Minimum 2 GPUs required (1 for rendering, 1 for inference)
  • CPU: 32 (minimum requirement)
  • RAM: 128 GB (minimum requirement)
  • NVIDIA GPU architecture: Ampere and beyond
• Software:
  • Driver:
    • Recommended: 525.85.05 (GameReady, Studio, RTX/Quadro), 525.85.12 (Grid/vGPU)
    • Minimum: 470.121
  • Kubernetes or MicroK8s installed
• Storage backend
  • Amazon S3
  • Enterprise Nucleus