USD Search API

Homepage: https://github.com/NVIDIA-omniverse/usd-search

Requirements

Overview

USD Search API is a collection of cloud-native microservices that enable developers, creators, and workflow specialists to efficiently search through vast collections of OpenUSD data, images, and other assets using natural language or image-based inputs.

With these production-ready microservices, developers can deploy USD Search API onto their own infrastructure. With USD Search API's artificial intelligence (AI) features, you can quickly locate untagged and unstructured 3D data and digital assets, saving time navigating unstructured, untagged 3D data. USD Search API is capable of searching and indexing 3D asset databases, as well as navigating complex 3D scenes to perform spatial searches, without requiring manual tagging of assets.

The document describes how USD Search API helm chart could be configured and installed on a kubernetes cluster and connected to a wide range of storage backends.

NOTE: USD Search API helm chart assumes that a Kubernetes cluster is already available and configured. Further, depending on the ingress controller and namespace, some changes may be required.

Table of Contents

• Prerequisites
• Packages and Access
• Deployment
  • AWS S3
    • Alternative installation options
    • Additional parameters: re-scan frequency
  • S3proxy
    • Examples for Different Storage Backends
      • Any S3 compatible Storage Backend
      • Azure Blob Storage Backend
      • Google Cloud Storage Backend
    • Alternative installation options
    • Extra Java options
  • Omniverse Nucleus Server
    • Alternative installation options
      • Nucleus API token
    • USD Search REST API Authentication
      • Admin Access Key
      • Disable Access verification
  • Omniverse Storage APIs
    • SSL support
    • Authentication
    • Alternative installation options
  • API endpoint access
  • Monitoring
• Post-installation
  • Deployment status check
  • Testing
• Uninstall
  • Rendering jobs clean-up
  • Secrets and persistent volumes clean-up
• Experimental
  • Sample Explorer WebUI
  • Admin tools (beta)
    • Re-indexing
    • Storage backend listing
  • Rendering service
    • Authentication
• Advanced configuration
  • Indexing path filtering
  • Thumbnail indexing settings
  • Plugins
    • Horizontal Pod Autoscaler
    • Concurrent processing
  • Rendering Job configuration
    • Number of parallel rendering jobs
    • Number of parallel rendering workers per rendering job
    • Rendering Job Timeout
    • Additional configuration settings
      • Annotations
      • Tolerations
    • Resources
  • Persistence (experimental)
  • USD properties search
  • VLM-based automatic captioning and tagging
    • LLM/VLM provider
    • Customization
      • Image prompt configuration
      • Metadata fields configuration
  • VLM-based verification of results with respect to input query
    • LLM/VLM provider
  • Search Backend configuration
    • External OpenSearch instance
    • OpenSearch authentication
  • OTEL Telemetry and Traces collection
    • Trace collection
    • Search telemetry collection
  • Values
    • Global settings
  • Embedding service settings
  • Asset Graph Service additional settings
  • Other settings
  • Redis instance settings
  • OpenSearch instance settings
  • Neo4j instance settings
  • Maintainers
• License
• FAQ
  • USD assets organization best practices
  • Image-based search best practices
  • Embedding service falls back to CPU with the GPU Operator installed
  • Redis Persistent Volume Claim
  • Microk8s CA certificate issues
  • Redis CrashLoopBackOff
  • OpenSearch Persistent Volume Claim
  • Incorrect Service Registration Token with Omniverse Nucleus storage backend
  • OpenSearch Virtual Memory (vm.max_map_count)
  • Storage backend connection
  • Search speed improvement
    • Increase the number of OpenSearch replicas
  • Indexing speed improvement
    • Enable shader caching in Rendering jobs
    • Scale the cluster
  • Metrics are missing in Grafana with monitoring enabled
• Get Help

Prerequisites

Kubernetes cluster with the following features enabled:

• Role-based access control (RBAC) - required for creating GPU-based asset rendering jobs
• NVIDIA k8s device plugin - required for execution of asset rendering jobs. Alternatively, NVIDIA GPU-operator (which includes NVIDIA k8s device plugin) could be used.

Optional:

• Metrics server - required for Horizontal Pod Autoscaling (HPA)
• Prometheus Stack - required for service monitoring
• Dynamic Persistent Volume provisioning - can be utilized by dependent helm charts (e.g. OpenSearch, Redis, Neo4j) in order to automatically provision required persistent volumes

Helm version is higher than 3.0.0. To check this run the following command

helm version
version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}

Packages and Access

Generate your NGC helm and container registry API Key prior to fetch helm chart. See onboarding guide here.

Fetch the latest helm chart from the registry

helm fetch https://helm.ngc.nvidia.com/nvidia/usdsearch/charts/usdsearch-1.4.0.tgz \
	--username='$oauthtoken' \
	--password=<YOUR API KEY>

Deployment

USD Search can be deployed to index data from an AWS S3 bucket, an Omniverse Nucleus server or a broad range of other backends by utilizing S3proxy functionality. Detailed deployment commands can be found in the sections below.

AWS S3

To deploy the USD Search Helm chart and connect it to an AWS S3 bucket for indexing, run the following command:

helm install <deployment name> usdsearch-1.4.0.tgz \
  --set global.accept_eula=true \
  --set global.storage_backend_type=s3 \
  --set global.s3.bucket_name=<S3 bucket name> \
  --set global.s3.region_name=<S3 bucket region>  \
  --set global.s3.aws_access_key_id=<AWS access key ID> \
  --set global.s3.aws_secret_access_key=<AWS secret access key>  \
  --set global.secrets.create.auth=true \
  --set global.secrets.create.registry=true \
  --set global.ngcAPIKey=<NGC API KEY> \
  --set api-gateway.image.pullSecrets={nvcr.io}

NOTE: Please refer to NGC documentation for generating the NGC API Key.

Alternative installation options

The above command creates all required secrets on helm chart deployment, which is done for convenience. It may, however, be not acceptable in some deployment scenarios to pass credential information through helm chart values. To address this issue it is possible to create kubernetes secret with access information manually with the following command:

kubectl create secret generic deepsearch-s3-credentials \
  --from-literal=AWS_ACCESS_KEY_ID=<AWS access key ID> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<AWS secret access key>

Similarly, it is possible to create a registry access secret with the following command:

kubectl create secret docker-registry nvcr.io \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password=<NGC API KEY> \
  --docker-email=<your-email>

With the above secrets created, the helm chart installation command can be simplified to:

helm install <deployment name> usdsearch-1.4.0.tgz \
  --set global.accept_eula=true \
  --set global.storage_backend_type=s3 \
  --set global.s3.bucket_name=<S3 bucket name> \
  --set global.s3.region_name=<S3 bucket region>  \
  --set api-gateway.image.pullSecrets={nvcr.io} \
  --set global.imagePullSecrets={nvcr.io}

Additional parameters: re-scan frequency

USD Search re-scans the S3 bucket daily to detect new data while minimizing load. You can adjust the re-scan frequency by adding the following argument to the installation or upgrade command:

  --set global.s3.re_scan_timeout=<re scan timeout time in seconds>

NOTE: Setting this parameter too low may cause processing queues to grow faster than your USD Search instance can handle. It is recommended to monitor the processing queues in Grafana and adjust the parameter to prevent excessive queue growth.

S3proxy

USD Search supports integration with an S3-compatible proxy, such as S3proxy, for indexing. By default, no specific image is configured for S3proxy (image.repository.url is unset), allowing flexibility for users to choose their preferred image. However, the official andrewgaul/s3proxy image is recommended, as it is Apache-licensed, regularly updated.

To deploy the USD Search Helm chart with S3proxy, ensure that the s3proxy feature is enabled and configured in the values.yaml file or through Helm overrides. Below you will find commands to install S3proxy with different backends.

For more details on configuring different storage backends, refer to the official S3proxy documentation or consult the cloud provider documentation for AWS S3, Azure Blob Storage, and Google Cloud Storage.

Examples for Different Storage Backends

Any S3 compatible Storage Backend

Using S3proxy functionality is it possible to seamlessly connect to any S3 compatible storage backends (e.g. Dell ObjectScale, DreamObjects, etc.)

helm upgrade <deployment name> usdsearch-1.4.0.tgz --install \
  --namespace <namespace> \
  --set global.s3proxy.enabled=true \
  --set global.accept_eula=true \
  --set global.s3.bucket_name=<your-bucket-name> \
  --set s3proxy.image.url=docker.io/andrewgaul/s3proxy:latest \
  --set s3proxy.jclouds.provider="s3" \
  --set s3proxy.jclouds.endpoint=<your-storage-backend-endpoint> \
  --set s3proxy.jclouds.identity=<your-storage-backend-identity> \
  --set s3proxy.jclouds.credential=<your-storage-backend-key> \
  --set global.ngcAPIKey=<NGC-API-KEY> \
  --set global.secrets.create.registry=true \
  --set global.secrets.create.auth=true \
  --set api-gateway.image.pullSecrets={nvcr.io}

Azure Blob Storage Backend

Using S3proxy functionality is it possible to seamlessly connect to Azure Blob Storage.

helm upgrade <deployment name> usdsearch-1.4.0.tgz --install \
  --namespace <namespace> \
  --set global.s3proxy.enabled=true \
  --set global.accept_eula=true \
  --set s3proxy.image.url=docker.io/andrewgaul/s3proxy:latest \
  --set global.s3.bucket_name=<your-bucket-name> \
  --set s3proxy.jclouds.provider="azureblob-sdk" \
  --set s3proxy.jclouds.endpoint="https://<azure-storage-account-name>.blob.core.windows.net/" \
  --set s3proxy.jclouds.identity=<your-storage-backend-identity> \
  --set s3proxy.jclouds.credential=<your-storage-backend-key> \
  --set global.ngcAPIKey=<NGC-API-KEY> \
  --set global.secrets.create.registry=true \
  --set global.secrets.create.auth=true \
  --set api-gateway.image.pullSecrets={nvcr.io}

Google Cloud Storage Backend

Using S3proxy functionality is it possible to seamlessly connect to Google Cloud Storage.

helm upgrade <deployment name> usdsearch-1.4.0.tgz --install \
  --namespace <namespace> \
  --set global.s3proxy.enabled=true \
  --set global.accept_eula=true \
  --set global.s3.bucket_name=<your-gcs-bucket-name> \
  --set s3proxy.image.url=docker.io/andrewgaul/s3proxy:latest \
  --set s3proxy.jclouds.provider="google-cloud-storage" \
  --set s3proxy.jclouds.endpoint="https://storage.googleapis.com" \
  --set s3proxy.jclouds.identity=<your-storage-backend-identity> \
  --set s3proxy.jclouds.credential=<your-storage-backend-key> \
  --set global.ngcAPIKey=<NGC-API-KEY> \
  --set global.secrets.create.registry=true \
  --set global.secrets.create.auth=true \
  --set api-gateway.image.pullSecrets={nvcr.io}

Alternative installation options

Sample installation commands illustrated in the above sections create all required secrets on helm chart deployment, which is done for convenience. It may, however, be not acceptable in some deployment scenarios to pass credential information through helm chart values. To address this issue it is possible to create kubernetes secret with access information manually with the following command:

kubectl create secret generic s3proxy-credentials \
  --from-literal=jclouds-identity=<your-storage-backend-identity> \
  --from-literal=jclouds-credential=<your-storage-backend-key> \
  --namespace <namespace>

It is also possible to create a registry access secret with the following command:

kubectl create secret docker-registry nvcr.io \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password=<NGC API KEY> \
  --docker-email=<your-email> \
  --namespace <namespace>

With the above secrets created, the helm chart installation command can be simplified to:

helm install <deployment name> usdsearch-1.4.0.tgz \
  --namespace <namespace> \
  --set global.s3proxy.enabled=true \
  --set s3proxy.image.url=docker.io/andrewgaul/s3proxy:latest \
  --set global.accept_eula=true \
  --set global.storage_backend_type=s3 \
  --set global.s3.bucket_name=<your bucket name (depending on the provider)> \
  --set s3proxy.jclouds.provider="<your-storage-backend-provider>" \
  --set s3proxy.jclouds.endpoint="<your-storage-backend-endpoint>" \
  --set api-gateway.image.pullSecrets={nvcr.io} \
  --set global.imagePullSecrets={nvcr.io}

Extra Java options

Additional Java options could be provided to the S3proxy container by setting the following value:

  --set s3proxy.extraJavaOpts=<additional java options>

For example in some cases it may be required to disable SSL verification by setting the following value:

  --set s3proxy.extraJavaOpts="-Djclouds.trust-all-certs=true -Djclouds.relax-hostname=true"

Omniverse Nucleus Server

USD Search API service requires a service account with administrator rights in order to index the content stored on the Nucleus server. It is possible to use the main Nucleus service account, generated during the Nucleus service installation time. Alternatively, it is possible to create a dedicated service account for USD Search. For the exact steps on how such account could be created please follow this guide.

When creating your own service account it is required to grant it admin access. This guide explains this process in detail.

After the service account's username and password are obtained, it is possible to deploy the USD Search Helm chart and connect it to an Omniverse Nucleus server for indexing, as follows:

helm install <deployment name> usdsearch-1.4.0.tgz \
 --set global.accept_eula=true \
 --set global.storage_backend_type=nucleus \
 --set global.nucleus.server=<Omniverse Nucleus server hostname or IP> \
 --set global.nucleus.username=<Omniverse service account name> \
 --set global.nucleus.password=<Omniverse service account password> \
 --set global.secrets.create.auth=true \
 --set global.secrets.create.registry=true \
 --set global.ngcAPIKey=<NGC API KEY> \
 --set api-gateway.image.pullSecrets={nvcr.io} \
 --set deepsearch-crawler.crawler.extraConfig.exclude_patterns={"omniverse://[^\/]+/NVIDIA.*"}

NOTE: Please refer to NGC documentation for generating the NGC API Key.

NOTE: The /NVIDIA sample data mount is excluded from indexing by default. To enable its indexing, remove the corresponding line from the command. You can also customize URL patterns for indexing, as explained in the Indexing path filtering section.

Alternative installation options

The above command creates all required secrets on helm chart deployment, which is done for convenience. It may, however, be not acceptable in some deployment scenarios to pass credential information through helm chart values. To address this issue it is possible to create kubernetes secret with access information manually with the following command:

kubectl create secret generic deepsearch-service-account \
  --from-literal=username=<Omniverse service account name> \
  --from-literal=password=<Omniverse service account password>

Similarly, it is possible to create a registry access secret with the following command:

kubectl create secret docker-registry nvcr.io \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password=<NGC API KEY> \
  --docker-email=<your-email>

With the above secrets created, the helm chart installation command can be simplified to:

helm install <deployment name> usdsearch-1.4.0.tgz \
  --set global.accept_eula=true \
  --set global.storage_backend_type=nucleus \
  --set global.nucleus.server=<Omniverse Nucleus server hostname or IP> \
  --set deepsearch-crawler.crawler.extraConfig.exclude_patterns={"omniverse://[^\/]+/NVIDIA.*"}
  --set api-gateway.image.pullSecrets={nvcr.io} \
  --set global.imagePullSecrets={nvcr.io}

Nucleus API token

Instead of using service account it is possible to rely on Nucleus API token. To do so, you need to input $omni-api-token as the username and the API token value as the password. Additionally you need to pass the following parameter to disable permissions check that is not required for Nucleus API tokens:

  --set global.nucleus.assert_admin_user='false'

USD Search REST API Authentication

By default, when connected to an Omniverse Nucleus server, search service verifies that the user has access to retrieved assets before returning them. Therefore, in order to access search functionality, it is required to provide with one of the following authentication methods:

• service account username / password pair

  • alternatively is it possible to rely on Nucleus API token and provide $omni-api-token as the username and the API token value as the password.

• Nucleus connection token that is generated by the client library.

Admin Access Key

At service deploy time it is possible to configure admin access key that allows by-passing this access verification step by setting the access_key configuration parameter as follows:

	--set ngsearch.microservices.search_rest_api.admin_authentication.access_key=<some key value>

when this field is left unset the access key will be auto-generated and stored in a configmap on the kubernetes cluster. To retrieve the value of this key you can run the following:

export NAMESPACE=<deployment namespace>
export HELM_NAME=<deployment name>
echo $(kubectl get cm $HELM_NAME-ngsearch-env-config -n $NAMESPACE -o "jsonpath={.data.DEEPSEARCH_BACKEND_ADMIN_ACCESS_KEY}")

Alternatively it is possible to run the following command:

helm status <deployment name>

which will show some information about the deployment and the list of useful commands. The value of Admin access key will be printed there as part of Accessing USD Search section.

Disable Access verification

If access verification is not desired it is possible to disable it by providing the following argument to the installation or upgrade command:

	--set ngsearch.microservices.search_rest_api.enable_access_verification=false

Omniverse Storage APIs

To deploy the USD Search Helm chart and connect it to an Omniverse Storage API for indexing, run the following command:

helm install <deployment name> usdsearch-1.4.0.tgz \
  --set global.accept_eula=true \
  --set global.storage_backend_type=storage_api \
  --set global.storage_api.grpc_endpoint=<Storage API gRPC endpoint> \
  --set global.storage_api.base_uri=<Storage API base URI> \
  --set global.secrets.create.registry=true \
  --set api-gateway.image.pullSecrets={nvcr.io} \
  --set global.imagePullSecrets={nvcr.io}

NOTE: Please refer to Omniverse Storage API documentation for more information about the Storage API.

SSL support

If the Storage API is configured to use SSL, set this to true. This is used to enable SSL for the Storage API connection.

  --set global.storage_api.ssl=true

Authentication

If Omniverse Storage APIs are configured to use authentication, it is possible to authenticate with the Storage API using either a token or OpenID. To do so, please set the following parameters:

• authentication.enabled=true
• authentication.type=token or authentication.type=openid
• authentication.token=
• authentication.openid.token_url=
• authentication.openid.client_id=
• authentication.openid.client_secret=

it is then required to add the following command line arguments during helm chart deployment for token authentication:

  --set global.secrets.create.auth=true
  --set global.storage_api.authentication.enabled=true \
  --set global.storage_api.authentication.type=token \
  --set global.storage_api.authentication.token=<Storage API token> \
  --set global.secrets.create.auth=true \

and for OpenID authentication:

  --set global.secrets.create.auth=true
  --set global.storage_api.authentication.enabled=true \
  --set global.storage_api.authentication.type=openid \
  --set global.storage_api.authentication.openid.token_url=<OpenID token URL> \
  --set global.storage_api.authentication.openid.client_id=<OpenID client ID> \
  --set global.storage_api.authentication.openid.client_secret=<OpenID client secret> \

Thumbnail retrieval

When using the Storage API backend, thumbnails are not resolved from the filesystem (the deepsearch.thumbnail_settings configuration does not apply). Instead, the service reads the thumbnail URL directly from a field in the asset's metadata.

By default the field named thumbnail_url is used. You can override this with an ordered list of field names; the first field present in the asset metadata whose value is a non-empty string is used as the thumbnail URL:

  --set global.storage_api.thumbnail_metadata_fields[0]=thumbnail_url

or via a values file:

global:
  storage_api:
    thumbnail_metadata_fields:
      - thumbnail_url

Alternative installation options

The above command creates all required secrets on helm chart deployment, which is done for convenience. It may, however, be not acceptable in some deployment scenarios to pass credential information through helm chart values. To address this issue it is possible to create kubernetes secret with access information manually with the following command:

kubectl create secret generic deepsearch-storage-api-credentials \
  --from-literal=token=<Storage API token>

Similarly, it is possible to create a registry access secret with the following command:

kubectl create secret docker-registry nvcr.io \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password=<NGC API KEY> \
  --docker-email=<your-email>

With the above secrets created, the helm chart installation command can be simplified to:

helm install <deployment name> usdsearch-1.4.0.tgz \
  --set global.accept_eula=true \
  --set global.storage_backend_type=storage_api \
  --set global.storage_api.grpc_endpoint=<Storage API gRPC endpoint> \
  --set global.storage_api.base_uri=<Storage API base URI> \
  --set global.storage_api.ssl=true \
  --set global.storage_api.authentication.enabled=true \
  --set global.storage_api.authentication.type=token \
  --set global.storage_api.authentication.secret_name=storage-api-credentials \
  --set global.storage_api.authentication.token=<Storage API token> \
  --set api-gateway.image.pullSecrets={nvcr.io} \
  --set global.imagePullSecrets={nvcr.io}

API endpoint access

All the USD Search functionality is unified under a single API gateway. API Gateway service configuration. By default the ClusterIP service type is used, however, it is possible to override this and make the endpoint publicly accessible. Please refer to NGINX helm chart service configuration for a complete list of settings.

Below you can find several sample configurations for the API gateway endpoint, depending on the desired service type, and instructions on how to access it:

• ClusterIP (default) - If the ClusterIP service type is used, you can access the API externally via port-forwarding from the Kubernetes cluster as described below. For internal access, use the service name directly.

  API Gateway port-forward example:

  kubectl port-forward -n <NAMESPACE> svc/<deployment name>-api-gateway 8080:80
  

  The endpoint should then be accessible at http://localhost:8080.

• NodePort - You can specify the NodePort service type to make the endpoint publicly accessible using the Kubernetes cluster’s external IP. Typically, NodePort values must be in the >30000 range. To enable this, use the following command line arguments:

  	--set api-gateway.service.type=NodePort \
  	--set api-gateway.service.nodePorts.http=<NodePort value>
  

  the service will then be accessible at http://<external cluster IP>:<NodePort value>

  NOTE: Please refer to Kubernetes documentation more information about NodePort type services.

• Ingress - You can enable Ingress to make the service available at a specified hostname. This requires an Ingress controller to be set up, and you may need to specify an Ingress class that matches your controller. Use the following command line arguments to configure it:

  	--set api-gateway.ingress.enabled=true \
  	--set api-gateway.ingress.hostname=<service hostname> \
      --set api-gateway.ingress.ingressClassName=<ingress class name, e.g. 'nginx'>
  

  the service will then be accessible at http://<service hostname>

Monitoring

The USD Search Helm chart includes monitoring functionality that sets up metrics collection and pre-configured dashboards for tracking background indexing progress and the overall system state. This requires the Prometheus Operator and Grafana with dashboard provisioning. You can use the Kube Prometheus Stack to meet these requirements. To enable monitoring, provide the following command line arguments:

	--set deployGrafanaDashboards=true \
	--set deployServiceMonitors=true

With the above flag enabled the metrics will be automatically scraped by Prometheus service (part of Prometheus stack) and visualized in Grafana as USD Search / Plugin Processing Dashboard and USD Search / Metadata Indexing and Crawler Dashboard dashboards.

Post-installation

Deployment status check

After the helm chart is installed you can check the state of the deployment as follows:

helm status <deployment name>

This command should print the following to the standard output:

NAME: <deployment name>
LAST DEPLOYED: <deployment time>
NAMESPACE: <deployment namespace>
STATUS: deployed
REVISION: 1
NOTES:
  ...

It will also show some useful commands for checking the following:

• General deployment information (e.g. deployment name and namespace)
• Storage backend information
• USD Search API service information
• USD Search API service access information
• Couple of useful commands to check pod status
• Clean-up commands configured specifically for the deployment

Testing

The first installation of deployment may take some time, as all the containers will be pulled from the registry. Subsequent installations, however, will be faster.

In order to verify that deployment is successful and the services are working as expected it is possible to run the following

helm test <deployment name>

For a successful deployment this should print the following to standard output:

...
TEST SUITE:     <deployment name>-ags-api-verification
Last Started:   Fri Nov  8 12:01:47 2024
Last Completed: Fri Nov  8 12:01:52 2024
Phase:          Succeeded
TEST SUITE:     <deployment name>-database-search-api-verification
Last Started:   Fri Nov  8 12:01:53 2024
Last Completed: Fri Nov  8 12:01:59 2024
Phase:          Succeeded
TEST SUITE:     <deployment name>-s3-storage-check
Last Started:   Fri Nov  8 12:01:59 2024
Last Completed: Fri Nov  8 12:02:10 2024
Phase:          Succeeded
...

In case some of these test return a Failed status, please refer to the following resources for debugging the deployment:

• Frequently asked questions section
• Kubernetes application debugging guide

Uninstall

To uninstall the USD Search API deployment, run the following command:

helm uninstall <deployment name>

Rendering jobs clean-up

Rendering jobs created during deployment are not automatically removed on helm uninstall and may occupy resources on the cluster. In order to remove them run the following command:

kubectl delete jobs \
	-l 'app.kubernetes.io/instance=<deployment name>,deepsearch.job-type=rendering' \
	--field-selector status.successful=0

This command only removes those jobs are running or failed execution. Successfully terminated jobs do not occupy any resources on the kubernetes cluster and would be automatically removed after 1 hour, so it is not necessary to clean them up. Nevertheless, in some cases it may be preferred to remove those jobs as well, which could be done with the following command:

kubectl delete jobs \
	-l 'app.kubernetes.io/instance=<deployment name>,deepsearch.job-type=rendering' \
	--field-selector status.successful=1

Secrets and persistent volumes clean-up

Secrets and persistent volume claims (PVCs) are not automatically removed on helm uninstall. They are retained in case the instance needs to be re-created. In some cases it could be desired to remove them as well (e.g. in case a clean install from scratch is desired). The following set of commands could be used to achieve this:

• Secrets:

kubectl delete secrets $(kubectl get secrets \
	-l 'app.kubernetes.io/instance=<deployment name>' \
	-o custom-columns=":metadata.name" | awk '{print}' ORS=' ')

• Persistent volume claims (PVCs):

kubectl delete pvc $(kubectl get pvc \
	-l 'app.kubernetes.io/instance=<deployment name>' \
	-o custom-columns=":metadata.name" | awk '{print}' ORS=' ')
kubectl delete pvc data-neo4j-0

Experimental

Sample Explorer WebUI

In order to quickly get started and experiment with USD Search APIs we provide a sample explorer web app that illustrates a way how USD Search APIs could be accessed.

In you order to enable it you need to run the following command to generate WebUI setup configuration:

echo 'deepsearch_explorer_deployment:
  enabled: true

deepsearch-explorer:
  image:
    registry: docker.io
    repository: bitnamilegacy/nginx
    tag: latest

  initContainers:
    - name: git-clone-and-build
      image: node:18-alpine
      command:
        - /bin/sh
        - -c
      args:
        - |
          apk add --no-cache git
          git clone -b ${GIT_REF} ${GIT_REPO_URL} /source && cd /source && cd ${SUB_PATH}
          npm install && npm ci --production=false && npm run build
          cp -r build/* /app/
      env:
        - name: GIT_REPO_URL
          value: "https://github.com/NVIDIA-Omniverse/usdsearch-client.git"
        - name: GIT_REF
          value: "main"
        - name: SUB_PATH
          value: "./web"
      volumeMounts:
        - name: app-volume
          mountPath: /app

  extraVolumeMounts:
    - name: app-volume
      mountPath: /app
      readOnly: true

  extraVolumes:
    - name: app-volume
      emptyDir:
        medium: Memory' > sample-webui.yaml

Then you need to add this configuration to you helm installation command as follows:

helm install ... -f sample-webui.yaml

Admin tools (beta)

We have added some administrative tools that allow performing different actions with USD Search API. You can find the list of available tools below.

Re-indexing

Re-indexing allows you to trigger priority processing of the assets on the storage backend:

kubectl exec -it $(kubectl get pods -l deepsearch.service.name=info-endpoint -o jsonpath='{.items[0].metadata.name}') -- \
    python -m usdsearch.admin.tools reindex <path>

By default this would attempt re-indexing those assets from the given path that have non-Ok statuses.

In order to learn about additional configuration options that re-indexing tool supports please refer to its help, which can be accessed as follows:

kubectl exec -it $(kubectl get pods -l deepsearch.service.name=info-endpoint -o jsonpath='{.items[0].metadata.name}') -- \
    python -m usdsearch.admin.tools reindex --help

Storage backend listing

Storage backend listing allows you to see the content on the certain path on the storage backend. It can be executed as follows:

kubectl exec -it $(kubectl get pods -l deepsearch.service.name=info-endpoint -o jsonpath='{.items[0].metadata.name}') -- \
    python -m usdsearch.admin.tools ls <path>

In order to learn about additional configuration options that re-indexing tool supports please refer to its help, which can be accessed as follows:

kubectl exec -it $(kubectl get pods -l deepsearch.service.name=info-endpoint -o jsonpath='{.items[0].metadata.name}') -- \
    python -m usdsearch.admin.tools ls --help

Rendering service

Rendering service can be deployed as an alternative to creating rendering jobs on demand. This has the following advantages:

• allows service administrators to better control resource utilization.
• improves rendering throughput as rendering service caches both shader and rendering data throughout the lifetime of the rendering service pod.
• no need for RBAC creation as the service is created by helm chart (not by worker pods)

In order to deploy the rendering service, the following additional parameters need to be set:

  --set rendering_service_deployment.enabled=true \
  --set deepsearch.microservices.rendering_service.enabled=true \
  --set deepsearch.microservices.k8s_renderer.enabled=false \
  --set deepsearch.microservices.plugin_worker.rendering_settings.renderer_type=rendering_service \
  --set deepsearch.rbac.create=false

The rendering service can also be hosted externally (e.g. on NVCF) in which case the following parameters need to be set:

  --set rendering_service_deployment.enabled=false \
  --set deepsearch.microservices.rendering_service.settings.rendering_service_url=<rendering service URL>

Authentication

If the rendering service endpoint is protected with an API key - it can be provided via a kubernetes secret as follows:

  --set global.secrets.create.rendering_service=true \
  --set deepsearch.microservices.rendering_service.authentication.enabled=true \
  --set deepsearch.microservices.rendering_service.authentication.api_key=<rendering service API key> \

alternatively, it is possible to create the secret manually with the following command:

  kubectl create secret generic rendering-service-api-key-secret \
    --from-literal=api-key=<rendering service API key> \
    --namespace <namespace>

and then set the following parameter:

  --set global.secrets.create.rendering_service=false \
  --set deepsearch.microservices.rendering_service.authentication.enabled=true \
  --set deepsearch.microservices.rendering_service.authentication.api_key_secret_name=rendering-service-api-key-secret \
  --set deepsearch.microservices.rendering_service.authentication.api_key_secret_field=api-key

Advanced configuration

NOTE: Changing the settings provided below is typically not required for a standard installation of USD Search. These settings, however, allow customizing the deployment and tuning it for the particular use-case and infrastructure.

Below several additional parameters that allow customizing USD Search service are described. For convenience, instead of providing them all as command line arguments, it is recommended to pass them to the installation command as a configuration file as follows:

helm install .... -f my-usdsearch-config.yaml

where my-usdsearch-config.yaml file can have the following additional settings.

Indexing path filtering

By default USD Search indexes the all the assets that can be found on the server. It is, however, possible to explicitly define which URL patterns should be included in indexing and which should be excluded. The URL definition supports python regex syntax. In order to include / exclude some of the file patterns - it is possible to specify the following in the my-usdsearch-config.yaml file:

deepsearch-crawler:
  crawler:
    extraConfig:
      include_patterns:
      - <.*regexp of the folder that needs to be included.*>
      exclude_patterns:
      - <.*regexp of the folder that needs to be excluded.*>

Thumbnail indexing settings

Configuration for how the service locates thumbnail files for each asset.

There are two modes for specifying which files are considered thumbnails:

• Without filepath_patterns (default) — thumbnails are resolved using relative_location and suffixes. For each asset the service looks for files matching:

  {folder_name}/{relative_location}/256x256/{file_name}{suffix}.png
  

  where {folder_name} and {file_name} are derived from the original asset path, and each value in suffixes is tried in order. For example, the default configuration:

  thumbnail_settings:
    relative_location: ".thumbs"
    suffixes:
      - ""
      - ".auto"
  

  will search for both {folder_name}/.thumbs/256x256/{file_name}.png and {folder_name}/.thumbs/256x256/{file_name}.auto.png, and use all available thumbnails for indexing.

• With filepath_patterns — the service matches thumbnail files against the provided list of regex patterns instead. Each pattern may reference {folder_name} and {file_name} (populated from the original asset path at runtime). All patterns are tried and all matching thumbnails are used for indexing. For example:

  thumbnail_settings:
    filepath_patterns:
      # Numbered thumbnails: asset.png1, asset.png2, ...
      - "{folder_name}/\\.thumbs/256x256/{file_name}\\.png(?:\\d+)$"
      # Standard and auto thumbnails: asset.png, asset.auto.png
      - "{folder_name}/\\.thumbs/256x256/{file_name}(\\.auto)?\\.png$"
      # Thumbnails stored in a sibling folder named "previews"
      - "{folder_name}/previews/{file_name}\\.png$"
  

The following thumbnail file formats are supported: all formats supported by the PIL library and GIF. When a thumbnail is a GIF file, it will be split into multiple frames by sampling every Nth frame (default: every frame), capped at a maximum number of frames (default: 512), configurable per plugin via the gif_sampling_mode (fixed = every Nth frame, or uniform = gif_max_frames frames spread evenly across the GIF), gif_frame_sample_frequency, and gif_max_frames parameters of the thumbnail_to_embedding and thumbnail_to_vision_metadata plugins.

NOTE: These settings apply only to the Nucleus and S3 storage backends. When using the Storage API backend, thumbnails are not resolved from the filesystem. Instead, the thumbnail URL is read directly from the asset metadata. The metadata field (or fields) that hold the thumbnail URL are controlled by the global.storage_api.thumbnail_metadata_fields setting. Please refer to the Thumbnail retrieval section for more information.

Plugins

USD Search API service is built as a collection of plugins. These plugins are designed to focus on specific task that are outlined below:

• asset_graph_generation - extracts USD prim structure and metadata from USD assets

  • supports USD asset formats (.usd, .usda, .usdc, .usdz)
  • extracts all dependencies of a USD stage and stores it as a graph in Neo4j database
  • extracts asset properties, please refer to USD properties search section for more details on how the functionality could be configured for different use-cases.
  • enabled by default, if not required - could be disabled by providing the following command line argument during helm installation:
      --set deepsearch.plugins.asset_graph_generation.active=False
    

• image_to_embedding - extracts CLIP embeddings from image data

  • supports all the image file formats supported by PIL library and .exr file format
  • enabled by default, if not required - could be disabled by providing the following command line argument during helm installation:
      --set deepsearch.plugins.image_to_embedding.active=False
    

• image_to_vision_metadata - uses VLM configured in VLM-based automatic captioning and tagging section to automatically generate captions and tags from images

  • supports all the image file formats supported by PIL library and .exr file format
  • requires VLM to be properly configured, please refer to VLM-based automatic captioning and tagging section for more information
  • disabled by default, if not required - could be disabled by providing the following command line argument during helm installation:
      --set deepsearch.plugins.image_to_vision_metadata.active=True
    

• rendering_to_embedding - renders USD assets from multiple views and extracts CLIP embeddings from these renderings

  • supports USD asset formats (.usd, .usda, .usdc, .usdz)
  • enabled by default, if not required - could be disabled by providing the following command line argument during helm installation:
      --set deepsearch.plugins.rendering_to_embedding.active=False
    

• rendering_to_vision_metadata - renders USD assets from multiple views and uses VLM configured in VLM-based automatic captioning and tagging section to automatically generate captions and tags from renderings of these assets

  • supports USD asset formats (.usd, .usda, .usdc, .usdz)
  • requires VLM to be properly configured, please refer to VLM-based automatic captioning and tagging section for more information
  • disabled by default, if not required - could be disabled by providing the following command line argument during helm installation:
      --set deepsearch.plugins.rendering_to_vision_metadata.active=True
    

• thumbnail_generation - renders USD asset and uploads rendered image to the storage backend to serve as asset thumbnail

  • supports USD asset formats (.usd, .usda, .usdc, .usdz)
  • enabled by default, if not required - could be disabled by providing the following command line argument during helm installation:
      --set deepsearch.plugins.thumbnail_generation.active=False
    

• thumbnail_to_embedding - extracts CLIP embeddings from thumbnails of assets

  • any asset type is supported, provided this asset has a thumbnail
  • currently thumbnails are expected to be in the Omniverse Nucleus format:
    • the thumbnail images should be found in the .thumbs/256x256/ folder next to the asset.
    • they should have image should have either <original asset name>.png or <original asset name>.auto.png name.
  • enabled by default, if not required - could be disabled by providing the following command line argument during helm installation:
      --set deepsearch.plugins.thumbnail_to_embedding.active=False
    

• thumbnail_to_vision_metadata - uses VLM configured in VLM-based automatic captioning and tagging section to automatically generate captions and tags from thumbnails of assets

  • any asset type is supported, provided this asset has a thumbnail
  • currently thumbnails are expected to be in the Omniverse Nucleus format:
    • the thumbnail images should be found in the .thumbs/256x256/ folder next to the asset.
    • they should have image should have either <original asset name>.png or <original asset name>.auto.png name.
  • requires VLM to be properly configured, please refer to VLM-based automatic captioning and tagging section for more information
  • disabled by default, if not required - could be disabled by providing the following command line argument during helm installation:
      --set deepsearch.plugins.thumbnail_to_vision_metadata.active=True
    

Horizontal Pod Autoscaler

Each of the plugins is deployed as a separate k8s deployment that could be horizontally scaled. By default this functionality is disabled to avoid occupying to many resources. In order to enable it, one could add the following to individual plugin configuration.

hpa:
  enabled: true
  minReplicas: <desired minimum number of replicas (default 1) >
  maxReplicas: <desired maximum number of replicas (default 1) >
  targetCPUUtilizationPercentage: <desired target CPU utilization (default 80) >

for example for image_to_embedding plugin the configuration can look like this:

deepsearch:
  # ...
  plugins:
    # ...
    image_to_embedding:
      hpa:
        enabled: true
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilizationPercentage: 80

Concurrent processing

Some plugins may rely on external services and do not do a lot of processing themselves. In order to increase throughput it is possible to increase concurrency for each plugin by setting n_concurrent_queue_workers parameter to the desired value. For those plugins that require USD Asset rendering this value is set to 256 by default, others have this parameter set to 1.

Adjusting the value of this parameter may increase throughput and can be done as in the following example:

deepsearch:
  # ...
  plugins:
    # ...
    thumbnail_to_embedding:
      n_concurrent_queue_workers: 8

Rendering Job configuration

For rendering USD asset USD Search API rely on rendering jobs that are scheduled by Kubernetes on demand. If needed - rendering Job configuration could be adjusted.

Number of parallel rendering jobs

USD Search relies on kubernetes native scheduling functionality for scheduling rendering jobs depending on the availability of the resources on the kubernetes cluster. In some cases, however, (e.g. when working with Omniverse Nucleus server backend) it could be desired to limit the number of parallel rendering jobs created in order to control the load that is imposed on the storage backend.

To do so, one could use the maxRenderingJobsCount parameter, which could be provided to the helm installation command via the following command line argument:

 --set deepsearch.microservices.k8s_renderer.maxRenderingJobsCount=<max number of parallel rendering jobs>

or by modifying the my-usdsearch-config.yaml file as follows:

deepsearch:
  microservices:
    k8s_renderer:
      maxRenderingJobsCount: <max number of parallel rendering jobs>

If this parameter is left unset - no limit on the number of rendering jobs will be applied.

Number of parallel rendering workers per rendering job

USD relies on Omniverse Kit for rendering USD assets. By default only one Kit instance is created per GPU (rendering job). It is, however, possible to increase the number of parallel Kit workers that would process assets on a single GPU to improve throughput.

To do so, one could use the n_parallel_kit_workers parameter, which could be provided to the helm installation command via the following command line argument:

 --set deepsearch.microservices.k8s_renderer.n_parallel_kit_workers=<number of parallel Kit workers>

or by modifying the my-usdsearch-config.yaml file as follows:

deepsearch:
  microservices:
    k8s_renderer:
      n_parallel_kit_workers: <number of parallel Kit workers>

Rendering Job Timeout

By default, the rendering job timeout is set to 1 hour. This can be adjusted by setting the activeDeadlineSeconds parameter as follows:

 --set deepsearch.microservices.k8s_renderer.activeDeadlineSeconds=<max amount of time that is allocated for job execution>

or by modifying the my-usdsearch-config.yaml file as follows:

deepsearch:
  microservices:
    k8s_renderer:
      activeDeadlineSeconds: <max amount of time that is allocated for job execution>

Additional configuration settings

Annotations

Additional optional annotations of the Rendering Job pod can be configured by setting the render_job_pod_annotations parameter as follows:

 --set deepsearch.microservices.k8s_renderer.render_job_pod_annotations.<annotation name>=<annotation value>

or by modifying the my-usdsearch-config.yaml file as follows:

deepsearch:
  microservices:
    k8s_renderer:
      render_job_pod_annotations:
        <annotation name>: <annotation value>

If this parameter is left unset - no additional annotations will be applied to the Rendering Job pod.

Please refer to the Kubernetes documentation for more information on how to use annotations

Tolerations

Additional optional tolerations of the Rendering Job pod can be configured by modifying the my-usdsearch-config.yaml file as follows:

deepsearch:
  microservices:
    k8s_renderer:
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"

Please refer to the Kubernetes documentation for more information on how to use tolerations

Resources

Resource requests and limits for each individual rendering job could be adjusted by updating the resources parameter in the deepsearch.microservices.k8s_renderer section in my-usdsearch-config.yaml file. The default settings for resource requests and limits for the job are outlined below:

requests:
    memory: "30Gi"
    cpu: "4"
    nvidia.com/gpu: "1"
limits:
    memory: "30Gi"
    cpu: "11"
    nvidia.com/gpu: "1"

Persistence (experimental)

Rendering job is using Omniverse Kit to render USD assets. When doing rendering Omniverse Kit saves some information in cache to make processing more efficient. This information includes shader cache, which takes between 100 and 300 seconds to compute. By default cache it created as a memory volume that is preserved during the lifetime of the job and is removed after it has completed.

In order to optimize processing it is, however, possible to persist this cached information, so that subsequent runs of the job are faster. In order to achieve this it is required to appropriately configure deepsearch.persistence section.

For each of these cache locations there are several set-up options that are controlled by setting the type of cache:

• emptyDir: creates the cache volume that is available only through out the lifetime of the job. It does not keep cache information during between the runs, which reduces performance as every time shaders need to be re-complied and shader cache re-created.
• pvc: creates PVC to for caching the data, which allows it to be preserved between execution, which in turn improves the speed of sub-sequent rendering runs. When using pvc persistence type it is possible to provide a custom PVC name by setting persistentVolumeClaim.claimName accordingly. Alternatively, it is possible to set pvc.createPvc to true in which case PVC will be automatically created on demand.
  • NOTE: In case of a custom PVC setup - it is important to make sure it is set with accessModes: [ReadWriteMany] so that multiple workers are able to access it. Not all storage classes support this setting (e.g. in AWS EFS storage class is required), so it is important to make sure that an appropriate storage class is used.
• hostPath: uses local host storage to store cache information
  • NOTE: when using local storage, the target directory hostPath.path has to be manually created on the host machine.

NOTE: This is an experimental feature. Some functionality may not be supported in case of the heterogenous GPU setup.

USD properties search

Asset Graph Search (AGS) component allows searching various properties defined as part of USD files. By default all the properties that are have the semantic: prefix are indexed, however it is possible to customize this behavior as follows:

plugins:
  asset_graph_generation:
    indexed_property_prefixes: "<custom_prefix_1>:,<custom_prefix_2>:"

Here <custom_prefix_1>, <custom_prefix_2> are the prefixes of USD properties that should be indexed. One could provide multiple prefixes separated by commas.

Alternatively, it is possible to specify the list of property names that should be indexed as follows:

plugins:
  asset_graph_generation:
    indexed_properties: "<property_name1>,<property_name2>,<property_name3>"

Here <property_name1>,<property_name2>,<property_name3> is the list of property names that should be indexed. One could provide multiple property names separated by commas. If indexed_properties parameter remains unset - all the properties with the prefixes defined above are indexed.

NOTE: Only the properties from the default prims inside a USD stage are indexed and searchable.

VLM-based automatic captioning and tagging

LLM/VLM provider

Vision endpoint allows to automatically tag and caption various assets stored on the storage backend using an external Vision Language Model (VLM) service.

For more information about the available USD Search API plugins, please refer to Plugin Settings section.

All LLM/VLM access goes through ONE OpenAI-compatible provider. Set the endpoint once here (provider.base_url -> USDSEARCH_LLM_BASE_URL) and the key via the referenced secret (provider.api_key -> USDSEARCH_LLM_API_KEY). Point base_url at any OpenAI-API-compatible server. Examples: NVIDIA Inference Hub (default): base_url: "https://inference-api.nvidia.com" OpenAI: base_url: "https://api.openai.com/v1" Azure OpenAI: base_url: "https://.openai.azure.com/openai/deployments/" Self-hosted vLLM / LiteLLM: base_url: "http://my-vllm.internal:8000/v1"

Example — configure the provider at install time (the api_key lands in the usdsearch-llm-api-key-secret secret created by the chart):

helm install usdsearch ./helm/usdsearch
...
--set global.secrets.create.vlm=true
--set deepsearch.vision_endpoint.provider.api_key=
--set deepsearch.vision_endpoint.provider.base_url=
--set deepsearch.vision_endpoint.metadata_model=

Default provider settings:

api_key: ""
api_key_secret_field: api-key
api_key_secret_name: usdsearch-llm-api-key-secret
base_url: https://inference-api.nvidia.com
env_prefix: USDSEARCH_LLM_

Customization

It is possible to customize the type of information extracted by the VLM-based auto-captioning system. There are two ways how this can be achieved:

1. By providing a custom image prompt.
2. By providing a custom list of metadata fields.

Each of the above is described in the following sections.

Image prompt configuration

Image prompt provided to the VLM-based auto-captioning system is fully customizable and can be controlled by setting deepsearch.vision_endpoint.image_prompt parameter in the values.yaml file. Below is the default configuration for the image prompt, however it could be adjusted to better fit the needs of the specific use case.

NOTE: there are two fields defined in the prompt: {metadata_types} and {metadata_definitions}. Those get populated with the respective values defined in deepsearch.vision_endpoint.metadata_fields parameter.

You are an expert 3D asset cataloguer for USD Search, an asset library used across
synthetic data generation, robotics & simulation, industrial / warehouse / logistics,
autonomous driving, architecture, retail, digital twins, and games & film.

Analyze the provided rendered images of a single 3D asset and produce rich, accurate,
searchable metadata. Optimize for retrieval: imagine the many different users —
a robotics engineer, an SDG pipeline, a warehouse digital-twin team, a game artist —
and capture the terms each of them would search for.

## How to analyze
1. Identify the object as specifically as possible (not "vehicle" but "counterbalance forklift").
2. Note its function and which domains/pipelines it suits.
3. Read any visible text, branding, labels, or signage.
4. Note materials, dominant colors, real-world scale, condition, and any mechanical traits.
5. Brainstorm the keywords and synonyms a real user would type to find it.

## Output format (JSON object with exactly these fields)
{metadata_types}

## Field definitions
{metadata_definitions}

## Rules
- Output ONLY a valid JSON object with every field above — no prose, no markdown.
- Describe only what is actually visible. Never invent brands, text, or function.
- When a field doesn't apply or can't be determined, use "unknown" for single-value
  string fields and an empty list `[]` for list fields. Do not guess.
- Be specific over generic; prefer precise nouns and industry terms.
- Use lowercase for tags, object_type, use_cases, environment, colors, materials,
  physical_attributes (captions and verbatim visible_text keep natural casing).
- Pick exactly ONE value for category, style, scale, and state from their allowed lists.
- Tags are the most important field: be thorough, include synonyms and domain terms.
- Quality flags must be honest — they let users filter out broken assets.

Metadata fields configuration

VLM-based verification of results with respect to input query

VLM validation uses a Vision Language Model to verify that each search result visually matches the input query. It compares the query against the result's thumbnail and returns a match decision with confidence score and reasoning.

To enable validation of search results, set the following parameter to true:

    --set ngsearch.microservices.search_rest_api.validation.enabled=true

or specify ngsearch.microservices.search_rest_api.validation.enabled setting in the my-usdsearch-config.yaml file as follows:

ngsearch:
  microservices:
    search_rest_api:
      validation:
        enabled: true

To set the maximum number of concurrent requests to the VLM service, set the following parameter:

    --set ngsearch.microservices.search_rest_api.validation.max_concurrent_requests=<number of concurrent requests>

or specify ngsearch.microservices.search_rest_api.validation.max_concurrent_requests setting in the my-usdsearch-config.yaml file as follows:

ngsearch:
  microservices:
    search_rest_api:
      validation:
        max_concurrent_requests: <number of concurrent requests>

LLM/VLM provider

Per-result VLM validation runs on the same shared OpenAI-compatible provider (provider.api_key -> USDSEARCH_LLM_API_KEY, provider.base_url -> USDSEARCH_LLM_BASE_URL). The stack ships pointed at a default endpoint (NVIDIA Inference Hub); point base_url at any OpenAI-API-compatible server to use a different one.

Example — enable validation and configure the provider at install time:

helm install usdsearch ./helm/usdsearch
...
--set global.secrets.create.vlm=true
--set ngsearch.microservices.search_rest_api.validation.enabled=true
--set ngsearch.microservices.search_rest_api.validation.provider.api_key=
--set ngsearch.microservices.search_rest_api.validation.provider.base_url=
--set ngsearch.microservices.search_rest_api.validation.model=

Default provider settings:

api_key: ""
api_key_secret_field: api-key
api_key_secret_name: usdsearch-llm-api-key-secret
base_url: https://inference-api.nvidia.com
env_prefix: USDSEARCH_LLM_

Search Backend configuration

By default an instance of OpenSearch is deployed as part of the USD Search API helm chart. The following parameters can be set to configure the search backend:

• index_name - name of the search index prefix (default: my-usdsearch-instance-index). In practice there will be 2 indexes created in the Search Backend (OpenSearch):

  • <index_name>-ver5.0 - stores embedding data and is the main index, where search is being executed
  • <index_name>-ver4.0-image-cache - non-indexed data storage for assets' renderings.

• number_of_shards - number of shards of search indices (default: 3). Please refer to OpenSearch documentation for more information about the number of shards selection.

  NOTE: The number_of_shards value can only be set during the first installation of the USD Search API helm chart. If the index is already created on the OpenSearch instance - changing this parameter in the helm chart would have no effect. In order to modify this parameter after installation, you need to create a new OpenSearch index with the correct amount of shards and apply re-indexing as described in OpenSearch documentation.

External OpenSearch instance

If you wish to rely on your own instance of OpenSearch (external to the USD Search API helm chart), you can do the following:

• disable creation of the default OpenSearch instance by setting the following command line argument during helm chart deployment:

  --set opensearch_deployment.enabled=false

• set the following parameters:
  • host - hostname of the OpenSearch instance (default: deepsearch-opensearch-cluster-master)
  • port - port of the OpenSearch instance (default: 9200)
  • schema - schema of the OpenSearch instance (default: http)
  • use_ssl - whether to use SSL for the OpenSearch instance (default: false)
  • auth_secret_name - name of the secret that stores the authentication information for the OpenSearch instance. Please refer to OpenSearch authentication section for more information.
  • hosts - comma-separated list of additional OpenSearch instances to use for search (default: [])

OpenSearch authentication

In order to authenticate with OpenSearch cluster a kubernetes secret that stores a subset of the following parameters can be created:

• username
• password

The name of the secret can be specified by providing the following command line argument during helm chart deployment:

  --set global.search_backend_config.auth_secret_name=<secret-name>

Please refer to OpenSearch authentication for more information about different authentication possibilities within OpenSearch. If any of these parameters are not required for the chosen authentication method, they don't need to be set within a secret.

NOTE: By default there is no authentication enabled on the test OpenSearch instance that can be deployed with the USD Search API helm chart. Therefore, when using the test installation a secret with these parameters does not need to be created and the following variable can be left unchanged. Authentication on the test OpenSearch instance can be enabled if one needs it, as the test OpenSearch endpoint is exposed on the internal k8s network, so if there are other services running in the cluster they can also access it.

OTEL Telemetry and Traces collection

Trace collection

By default trace collection is disabled. Is it optionally possible to enable trace collection for USD Search REST API and AGS services. In order to do so, please set OTEL_SDK_DISABLED to "false" and OTEL_TRACES_EXPORTER to "true" and provide a valid OTEL_EXPORTER_OTLP_ENDPOINT URL as follows:

  --set global.tracing.OTEL_SDK_DISABLED=false \
  --set global.tracing.OTEL_TRACES_EXPORTER=true \
  --set global.tracing.OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4318

Search telemetry collection

Search service has the possibility to gather telemetry information about searches executed by the users of the system. This information can then be used to understand what queries are most frequently executed and also allow to track down issues if inconsistency appears in the search results.

Telemetry information includes:

• Overall duration of request processing
• Time spent waiting for response from the Search backend
• Parsed version of the input query that is then converted to a request to OpenSearch service

NOTE: that no information about the user is stored in the system.

By default telemetry logging is switched off. In order to enable telemetry collection, please set the following command line parameters during helm chart deployment:

    --set global.tracing.OTEL_SDK_DISABLED=false \
    --set ngsearch.microservices.search_rest_api.use_search_telemetry=true \
    --set ngsearch.microservices.search_rest_api.search_telemetry_stdout=true

You will then be able to see telemetry information being printed in the logs of the REST API service.

Values

For convenience, configurable values of this Helm Chart are outlined in the sections below.

Global settings

false

"1.4.0"

searches: []

endpoint: ""
triton_server:
    ssl:
        enabled: false
    headers: {}
authentication:
    enabled: false
    token:
    secret_key: token
    secret_name: embedding-service-secret

false

pullPolicy: IfNotPresent
usdsearch:
    repository: usdsearch
    tag: ""
siglip2_triton:
    repository: siglip2-triton
    tag: ""
rendering_job:
    repository: usdsearch-kit-workflows
    tag: ""

null

null

"ngc-api"

"nvcr.io"

""

"nvcr.io/nvidia/usdsearch"

{
  "annotations": {
    "helm.sh/hook": "pre-install,pre-upgrade",
    "helm.sh/hook-weight": "-5",
    "helm.sh/resource-policy": "keep"
  },
  "create": {
    "auth": false,
    "embedding": false,
    "explorer_ui": false,
    "ngc_api": false,
    "registry": false,
    "rendering_service": false,
    "vlm": false
  }
}

"s3"

Embedding service settings

The following parameters describe embedding service configuration settings:

{}

1

requests:
    nvidia.com/gpu: 1
    cpu: 2
    memory: 7Gi
limits:
    nvidia.com/gpu: 1
    cpu: 4
    memory: 15Gi

""

emptyDir:
    medium: Memory
    sizeLimit: 256Mi

[]

Asset Graph Service additional settings

The following parameters allow configuring the Asset Graph Search (AGS) deployment:

5

""

true

Other settings

true

true

requests:
    cpu: 1
    memory: 10Gi
limits:
    cpu: 1
    memory: 10Gi

{
  "replicas": 1
}

replicas: 1

relative_location: ".thumbs"
suffixes:
    - ""
    - ".auto"
filepath_patterns:

enable: false

{
  "replicas": 1
}

64

true

enabled: false
maxReplicas: 5
minReplicas: 1
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 90

    --set ngsearch.microservices.search_rest_api.hpa.enabled=true

    --set ngsearch.microservices.search_rest_api.hpa.maxReplicas=<number of replicas>

    --set ngsearch.microservices.search_rest_api.hpa.minReplicas=<number of replicas>

system prompt + filter catalog shipped in the app image, reproduced inline below

    --set ngsearch.microservices.search_rest_api.llm_parsing.enabled=true

ngsearch:
  microservices:
    search_rest_api:
      llm_parsing:
        enabled: true

    --set ngsearch.microservices.search_rest_api.llm_parsing.model=<model id>

ngsearch:
  microservices:
    search_rest_api:
      llm_parsing:
        model: <model id>

    --set ngsearch.microservices.search_rest_api.llm_parsing.max_tokens=2048

false

{
  "replicas": 1
}

{
  "replicas": 1
}

{
  "replicas": 1
}

enabled: false

Redis instance settings

Configuration for the default Redis instance, which gets deployed when redis_deployment.enabled is set to true.

"standalone"

enabled: False

redis.commonConfiguration: |
  appendonly yes
  save ""
  databases 32

"bitnamilegacy/redis"

disableCommands: []
persistence:
    enabled: True
    size: 64Gi
resources:
    limits:
        memory: 10Gi
        ephemeral-storage: 2Gi
        cpu: 1000m

replicaCount: 0

enabled: true

OpenSearch instance settings

Configuration for the default OpenSearch cluster, which gets deployed when opensearch_deployment.enabled is set to true.

"server.name: dashboards\nserver.host: \"0.0.0.0\"\nserver.ssl.enabled: false\nopensearch.ssl.verificationMode: none\nopensearch.username: kibanaserver\nopensearch.password: kibanaserver\nopensearch.requestHeadersWhitelist: [authorization, securitytenant]\nopensearch.hosts: [\"http://deepsearch-opensearch-cluster-master:9200\"]\n"

"DISABLE_SECURITY_DASHBOARDS_PLUGIN"

"true"

{}

false

""

""

""

"/"

"nginx"

{}

"http://deepsearch-opensearch-cluster-master:9200"

"deepsearch-opensearch-cluster"

opensearch.config."opensearch.yml": |
  network.host: 0.0.0.0
  #knn.algo_param.index_thread_qty: 8
  plugins:
    security:
      disabled: true

"DISABLE_INSTALL_DEMO_CONFIG"

"true"

"deepsearch-opensearch-cluster-master"

"-Xmx4096M -Xms4096M"

enabled: true
size: 100Gi

3

requests:
    cpu: "2"
    memory: "8Gi"
limits:
    cpu: "3"
    memory: "8Gi"

enabled: false

{
  "enabled": false
}

false

enabled: true

  --set opensearch_deployment.enabled=false

Neo4j instance settings

Configuration for the default Neo4j instance, which gets deployed when neo4j_deployment.enabled is set to true.

server.config.strict_validation.enabled: "false"
server.memory.heap.initial_size: "8000m"
server.memory.heap.max_size: "8000m"

"[\"graph-data-science\", \"apoc\"]"

"neo4j"

"/bin/sh"

"-c"

"USER=$(cat /config/neo4j-auth/NEO4J_AUTH | cut -d'/' -f1)\nPASS=$(cat /config/neo4j-auth/NEO4J_AUTH | cut -d'/' -f2)\ncypher-shell --non-interactive -u \"$USER\" -p \"$PASS\" \"RETURN 1\" || exit 1\n"

name: neo4j
password: "password"
resources:
    requests:
        cpu: "4000m"
        memory: "14Gi"
    limits:
        cpu: "4000m"
        memory: "14Gi"

false

neo4j:
    enabled: true
    annotations: {}
    spec:
        type: ClusterIP

"ReadWriteOnce"

"100Gi"

"defaultStorageClass"

true

Maintainers

License

GOVERNING TERMS:

If you download the software and materials as available from the NVIDIA AI product portfolio, use is governed by the NVIDIA Software License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and the Product-Specific Terms for NVIDIA AI Products (found at https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/); except for the model which is governed by the NVIDIA AI Foundation Models Community License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-ai-foundation-models-community-license-agreement/.

If you download the software and materials as available from the NVIDIA Omniverse portfolio, use is governed by the NVIDIA Software License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and the Product-Specific Terms for NVIDIA Omniverse (found at NVIDIA Agreements | Enterprise Software | Product Specific Terms for Omniverse); except for the model which is governed by the NVIDIA AI Foundation Models Community License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-ai-foundation-models-community-license-agreement/.

FAQ

USD assets organization best practices

USD Search API service indexes individual assets that are stored on the storage backend. As described in Plugins section multiple asset formats are supported.

When it comes to USD assets, USD Search API service operates on the file level, so the whole USD asset / scene is indexed as a whole. If one has a large USD scene with multiple smaller prims baked in, only the overarching USD scene will be indexed as it would be the only actual file stored on the storage backend. Therefore, in case finding smaller objects that the scene is composed of is desired (e.g. in the case of doing in-scene search or finding scenes that contain specific objects), those individual objects should be:

• stored on the storage backend as individual USD assets,
• referenced in the main scene, instead of being baked into it.

Image-based search best practices

USD Search API relies on the SigLIP2 model to extract embedding from images. SigLIP2 model operates on RGB images of shape 384x384, therefore to achieve the best accuracy the input images should be squared and have at least 384x384 size.

If images have different dimensions - they will be rescaled such that the minimum of height and width is equal to 384 and then center-cropping will be applied to make sure aspect ratio is preserved and the SigLIP2 model receives a squared input.

Embedding service falls back to CPU with the GPU Operator installed

If the embedding service container starts on CPU even though the NVIDIA GPU Operator is installed on the cluster, the two most common causes are:

1. NVIDIA GPU driver version mismatch. The driver must be version 595.58.03 or higher. Verify the installed driver and upgrade it if it is older.

2. Multiple RuntimeClasses defined on the cluster. When the driver is correct, the GPU may not be exposed because more than one RuntimeClass is defined and the wrong one is selected by default. List the available RuntimeClasses with:

   kubectl get runtimeclass
   

   If several are present, override the default one for the embedding pod by setting the deepsearch.microservices.embedding.runtimeClassName parameter to the GPU RuntimeClass (e.g. nvidia):

     --set deepsearch.microservices.embedding.runtimeClassName=nvidia
   

Redis Persistent Volume Claim

USD Search API relies on Redis for all internal caching and uses the official Redis Helm chart for installation, which itself relies on the Persistent Volume Claim mechanism. For the Redis installation provided with USD Search API, a Persistent Volume is required that would be then claimed by Redis. An example configuration is shown below for a Persistent Volume that uses storage on the local file system.

apiVersion: v1
kind: PersistentVolume
metadata:
name: sample-pv-name
spec:
accessModes:
- ReadWriteOnce
capacity:
  storage: 100Gi
local:
  path: /var/lib/omni/volumes/001
nodeAffinity:
  required:
    nodeSelectorTerms:
    - matchExpressions:
    - key: kubernetes.io/hostname
    operator: In
    values:
    - "node-name"
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem

For more information on different types of Persistent Volumes and their setup procedures, please refer to the official Kubernetes documentation.

Microk8s CA certificate issues

When using a microk8s cluster it may happen that some pods (e.g. -deepsearch-worker-thumb-gen-bg-...) would crash with the following error:

  File "/usr/local/lib/python3.13/site-packages/monitor/src/monitor_worker.py", line 640, in task_processor
    raise Exception("Unexpected error: %s", str(exc)) from exc
Exception: ('Unexpected error: %s', "Cannot connect to host <some IP>:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: CA cert does not include key usage extension (_ssl.c:1032)')]")

This is a known issue with the microk8s server: https://github.com/canonical/microk8s/issues/4864.

Please refer to the fix suggested in this post to manually generate certificates and update your micork8s cluster.

NOTE: For a quick test it is possible to provide the following parameter to the helm chart deployment that disables SSL verification of CA kubernetes certificate:

  --set deepsearch.microservices.k8s_renderer.verify_k8s_ssl_cert=false

this, however, is strictly not recommended for production environments.

Redis CrashLoopBackOff

In rare cases it could happen that Redis appendonly file ends up in a corrupted state. In this case the following line will be printed in the logs of the redis pod:

Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix ....

This may happen if the node that is running redis got unexpectedly terminated or has run out of space.

In order to fix this issue, please execute the following set of commands.

NOTE: The steps below assume that there is only a single instance of USD Search API installed in a provided namespace. If that is not the case the way REDIS_STATEFULSET_NAME and REDIS_POD_NAME are computed need to be updated.

# prepare some settings
export NAMESPACE=<namespace where USD Search API is running>
export REDIS_AOF_FILE_NAME=<corrupted file name from the Redis log>
# get the name of the statefulset that is controlling Redis
export REDIS_STATEFULSET_NAME=$(kubectl get statefulset -n $NAMESPACE -o custom-columns=":metadata.name" | grep redis)
# get the name of the pod running Redis
export REDIS_POD_NAME=$(kubectl get pods -n $NAMESPACE -o custom-columns=":metadata.name" | grep redis)
# patch Redis statefulset to sleep (to exit crashbackloop)
kubectl patch statefulset -n $NAMESPACE $REDIS_STATEFULSET_NAME -p '{"spec": {"template": {"spec":{"containers":[{"name": "redis","args": ["-c", "sleep 1000000000"]}]}}}}'
# give k8s 5 seconds to restart redis
sleep 5
# fix corrupted redis file
kubectl exec -it -n $NAMESPACE $REDIS_POD_NAME -- redis-check-aof --fix /data/appendonlydir/$REDIS_AOF_FILE_NAME
# revert statefulset patching
kubectl patch statefulset -n $NAMESPACE $REDIS_STATEFULSET_NAME -p '{"spec": {"template": {"spec":{"containers":[{"name": "redis","args": ["-c", "/opt/bitnami/scripts/start-scripts/start-master.sh"]}]}}}}'
# delete running container to make sure it restarts
kubectl delete pods $REDIS_POD_NAME

OpenSearch Persistent Volume Claim

When using the instance of OpenSearch provided with the USD Search API helm chart, the official OpenSearch Helm chart will be used for installation. This Helm chart by default installs a 3-Node OpenSearch instance and requires Persistent Volume storage. Creation of Persistent Volumes can be done in the exactly same way as described in the previous section.

Incorrect Service Registration Token with Omniverse Nucleus storage backend

When configuring USD Search API, a failure to register with the Nucleus Discovery service will happen if the provided Nucleus registration token is incorrect. If this occurs, the following error may be displayed:

Deployment: internal registration failed: DENIED

To solve this issue, the correct service registration token needs to be provided and can be located in the following subfolder within the Nucleus Docker Compose installation location:

base_stack/secrets/svc_reg_token

OpenSearch Virtual Memory (vm.max_map_count)

On some systems, the value of the kernel parameter vm.max_map_count may be too low for OpenSearch. If this is the case, it is required to update the default value for vm.max_map_count to at least 262144, as described in the OpenSearch installation documentation.

To check the current value, run this command:

cat /proc/sys/vm/max_map_count

To increase the value, add the following line to /etc/sysctl.conf:

vm.max_map_count=262144

Then run the following to reload and apply the settings change.

sudo sysctl -p

Storage backend connection

Helm chart installation assumes that storage backend (AWS S3 bucket or Omniverse Nucleus Server) is available before installation and valid credential information is provided.

For convenience we have included a helm pre-installation hook that checks the backend connection before installing of the helm chart.

If storage backend is not available, then depending on the backend type, one of the following errors will be printed during execution of helm install command:

Error: INSTALLATION FAILED: failed pre-install: 1 error occurred:
	* job test-nucleus-storage-check failed: BackoffLimitExceeded

or

Error: INSTALLATION FAILED: failed pre-install: 1 error occurred:
	* job test-s3-storage-check failed: BackoffLimitExceeded

It could happen that connection with the storage backend is broken after helm chart is installed. This could occur if the storage backend is unreachable for some reason. In this case, you may notice that many pods enter the CrashLoopBackOff state. To confirm that the issue is indeed related to the storage backend connection, you can do one of the following:

1. run helm test which will verify storage backend connection as follows:

   helm test <deployment name>
   

2. check the logs of any pod that entered CrashLoopBackOff and if you see ConnectionError messages - that would mean that storage backend is for some reason unavailable.

Search speed improvement

In case slow search speeds are encountered, it is possible to do several optimizations from the helm chart level.

Increase the number of OpenSearch replicas

If the cluster permits - it is possible to increase the number of OpenSearch replicas that is used. By default the helm chart is set to use 3 replicas, which we found to be sufficient in our experiments, however, this parameter could be overwritten. Therefore, it is recommended to check opensearch.replicas setting in my-usdsearch-config.yaml and adjust it according the amount of available resources. Alternatively it is possible to set the desired number of OpenSearch replicas as a command line argument as follows:

	--set opensearch.replicas=<desired number of OpenSearch replicas>

Indexing speed improvement

In case slow indexing speeds are encountered, it is possible to do several optimizations from the helm chart level.

Enable shader caching in Rendering jobs

By default Rendering Jobs are using memory medium for shader cache, which is only available during the lifetime of a job and therefore such cache needs to be re-calculated for each rendering job, which adds a significant overhead. Please refer to Rendering Job configuration section for more information on how to setup persistence.

Scale the cluster

Rendering jobs only get allocated, when enough resources are available on the cluster. So adding a node with more GPUs will linearly increase indexing speed.

Metrics are missing in Grafana with monitoring enabled

It could happen that Prometheus metrics do not appear in Grafana installed by Kube Prometheus Stack. The most common reason for such behavior is that Prometheus operator may not be configured to monitor all namespaces on the kubernetes cluster. In order to let Prometheus monitor all namespaces set the following in Kube Prometheus stack configuration:

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector: {}

Get Help

Enterprise Support

Get access to knowledge base articles and support cases or submit a ticket.