USD Search API

Homepage: https://docs.omniverse.nvidia.com/services/latest/services/usd-search/overview.html

Requirements

Overview

USD Search API is a collection of cloud-native microservices that enable developers, creators, and workflow specialists to efficiently search through vast collections of OpenUSD data, images, and other assets using natural language or image-based inputs.

With these production-ready microservices, developers can deploy USD Search API onto their own infrastructure. With USD Search API's artificial intelligence (AI) features, you can quickly locate untagged and unstructured 3D data and digital assets, saving time navigating unstructured, untagged 3D data. USD Search API is capable of searching and indexing 3D asset databases, as well as navigating complex 3D scenes to perform spatial searches, without requiring manual tagging of assets.

The document describes how USD Search API helm chart could be configured and installed on a kubernetes cluster and connected to a wide range of storage backends.

NOTE: USD Search API helm chart assumes that a Kubernetes cluster is already available and configured. Further, depending on the ingress controller and namespace, some changes may be required.

Table of Contents

• Prerequisites
• Packages and Access
• Deployment
  • AWS S3
    • Alternative installation options
    • Additional parameters: re-scan frequency
  • S3proxy
    • Examples for Different Storage Backends
      • Azure Blob Storage Backend
      • Google Cloud Storage Backend
    • Documentation
  • Omniverse Nucleus Server
    • Alternative installation options
      • Nucleus API token
    • USD Search REST API Authentication
      • Admin Access Key
      • Disable Access verification
  • API endpoint access
  • Monitoring
• Post-installation
  • Deployment status check
  • Testing
• Uninstall
  • Rendering jobs clean-up
  • Secrets and persistent volumes clean-up
• Advanced configuration
  • Indexing path filtering
  • Plugins
    • Horizontal Pod Autoscaler
    • Concurrent processing
  • Rendering Job configuration
    • Limit the number of Kit rendering instances
    • Resources
  • Persistence (experimental)
  • USD properties search
  • VLM-based automatic captioning and tagging
    • VLM services
      • Anthropic
      • Azure OpenAI
      • Mistral AI
      • NVIDIA AI Endpoints
      • OpenAI
    • Customization
      • Image prompt configuration
      • Metadata fields configuration
  • Values
    • Global settings
  • Embedding service settings
  • Asset Graph Service additional settings
  • Other settings
  • Redis instance settings
  • OpenSearch instance settings
  • Neo4j instance settings
  • Maintainers
• License
• FAQ
  • USD assets organization best practices
  • Image-based search best practices
  • Redis Persistent Volume Claim
  • Redis CrashLoopBackOff
  • OpenSearch Persistent Volume Claim
  • Incorrect Service Registration Token with Omniverse Nucleus storage backend
  • OpenSearch Virtual Memory (vm.max_map_count)
  • Storage backend connection
  • Search speed improvement
    • Move embedding service to GPU
    • Increase the number of OpenSearch replicas
  • Indexing speed improvement
    • Enable shader caching in Rendering jobs
    • Scale the cluster
  • Metrics are missing in Grafana with monitoring enabled

Prerequisites

Kubernetes cluster with the following features enabled:

• Role-based access control (RBAC) - required for creating GPU-based asset rendering jobs
• NVIDIA k8s device plugin - required for execution of asset rendering jobs. Alternatively, NVIDIA GPU-operator (which includes NVIDIA k8s device plugin) could be used.

Optional:

• Metrics server - required for Horizontal Pod Autoscaling (HPA)
• Prometheus Stack - required for service monitoring
• Dynamic Persistent Volume provisioning - can be utilized by dependent helm charts (e.g. OpenSearch, Redis, Neo4j) in order to automatically provision required persistent volumes

Helm version is higher than 3.0.0. To check this run the following command

helm version
version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}

Packages and Access

Generate your NGC helm and container registry API Key prior to fetch helm chart. See onboarding guide here.

Fetch the latest helm chart from the registry

helm fetch https://helm.ngc.nvidia.com/nvidia/usdsearch/charts/usdsearch-1.1.1.tgz \
    --username='$oauthtoken' \
    --password=<YOUR API KEY>

Deployment

USD Search can be deployed to index data from an AWS S3 bucket, an Omniverse Nucleus server or a broad range of other backends by utilizing S3proxy functionality. Detailed deployment commands can be found in the sections below.

AWS S3

To deploy the USD Search Helm chart and connect it to an AWS S3 bucket for indexing, run the following command:

helm install <deployment name> usdsearch-1.1.1.tgz \
  --set global.accept_eula=true \
  --set global.storage_backend_type=s3 \
  --set global.s3.bucket_name=<S3 bucket name> \
  --set global.s3.region_name=<S3 bucket region>  \
  --set global.s3.aws_access_key_id=<AWS access key ID> \
  --set global.s3.aws_secret_access_key=<AWS secret access key>  \
  --set global.secrets.create.auth=true \
  --set global.secrets.create.registry=true \
  --set global.ngcAPIKey=<NGC API KEY> \
  --set api-gateway.image.pullSecrets={nvcr.io}

NOTE: Please refer to NGC documentation for generating the NGC API Key.

Alternative installation options

The above command creates all required secrets on helm chart deployment, which is done for convenience. It may, however, be not acceptable in some deployment scenarios to pass credential information through helm chart values. To address this issue it is possible to create kubernetes secret with access information manually with the following command:

kubectl create secret generic deepsearch-s3-credentials \
  --from-literal=AWS_ACCESS_KEY_ID=<AWS access key ID> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<AWS secret access key>

Similarly, it is possible to create a registry access secret with the following command:

kubectl create secret docker-registry nvcr.io \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password=<NGC API KEY> \
  --docker-email=<your-email>

With the above secrets created, the helm chart installation command can be simplified to:

helm install <deployment name> usdsearch-1.1.1.tgz \
  --set global.accept_eula=true \
  --set global.storage_backend_type=s3 \
  --set global.s3.bucket_name=<S3 bucket name> \
  --set global.s3.region_name=<S3 bucket region>  \
  --set api-gateway.image.pullSecrets={nvcr.io} \
  --set global.imagePullSecrets={nvcr.io}

Additional parameters: re-scan frequency

USD Search re-scans the S3 bucket daily to detect new data while minimizing load. You can adjust the re-scan frequency by adding the following argument to the installation or upgrade command:

  --set global.s3.re_scan_timeout=<re scan timeout time in seconds>

NOTE: Setting this parameter too low may cause processing queues to grow faster than your USD Search instance can handle. It is recommended to monitor the processing queues in Grafana and adjust the parameter to prevent excessive queue growth.

S3proxy

USD Search supports integration with an S3-compatible proxy, such as S3proxy, for indexing. By default, no specific image is configured for S3proxy (image.repository.url is unset), allowing flexibility for users to choose their preferred image. However, the official andrewgaul/s3proxy image is recommended, as it is Apache-licensed, regularly updated, and has been tested with USD Search API to ensure compatibility.

To deploy the USD Search Helm chart with S3proxy, ensure that the s3proxy feature is enabled and configured in the values.yaml file or through Helm overrides. Below you will find commands to install S3proxy with different backends.

Examples for Different Storage Backends

Before deploying S3Proxy with different storage backends such as Azure Blob or Google Cloud Storage, create the necessary Kubernetes secret to store the jclouds identity and credential:

kubectl create secret generic s3proxy-credentials \
  --from-literal=jclouds-identity=<storage-backend-identity> \
  --from-literal=jclouds-credential="<storage-backend-key>" \
  --namespace <namespace>

Replace <storage-backend-identity> and <storage-backend-key> with the corresponding values from your storage backend account.

You can verify that the secret is correctly created with:

kubectl get secret s3proxy-credentials -n <namespace> -o yaml

Azure Blob Storage Backend

helm upgrade <deployment name> usdsearch-1.1.1.tgz --install \
  --namespace <namespace> \
  --set global.s3proxy.enabled=true \
  --set global.accept_eula=true \
  --set s3proxy.image.url=docker.io/andrewgaul/s3proxy:latest \
  --set global.s3.bucket_name=<your-bucket-name> \
  --set s3proxy.jclouds.provider="azureblob-sdk" \
  --set s3proxy.jclouds.endpoint="https://<azure-storage-account-name>.blob.core.windows.net/" \
  --set global.ngcAPIKey=<NGC-API-KEY> \
  --set global.secrets.create.registry=true \
  --set api-gateway.image.pullSecrets={nvcr.io}

Google Cloud Storage Backend

helm upgrade <deployment name> usdsearch-1.1.1.tgz --install \
  --namespace <namespace> \
  --set global.s3proxy.enabled=true \
  --set global.accept_eula=true \
  --set global.s3.bucket_name=<your-gcs-bucket-name> \
  --set s3proxy.image.url=docker.io/andrewgaul/s3proxy:latest \
  --set s3proxy.jclouds.provider="google-cloud-storage" \
  --set s3proxy.jclouds.endpoint="https://storage.googleapis.com" \
  --set global.ngcAPIKey=<NGC-API-KEY> \
  --set global.secrets.create.registry=true \
  --set api-gateway.image.pullSecrets={nvcr.io}

Documentation

For more details on configuring different storage backends, refer to the official S3proxy documentation or consult the cloud provider documentation for AWS S3, Azure Blob Storage, and Google Cloud Storage.

Omniverse Nucleus Server

USD Search API service requires a service account with administrator rights in order to index the content stored on the Nucleus server. It is possible to use the main Nucleus service account, generated during the Nucleus service installation time. Alternatively, it is possible to create a dedicated service account for USD Search. For the exact steps on how such account could be created please follow this guide.

When creating your own service account it is required to grant it admin access. This guide explains this process in detail.

After the service account's username and password are obtained, it is possible to deploy the USD Search Helm chart and connect it to an Omniverse Nucleus server for indexing, as follows:

helm install <deployment name> usdsearch-1.1.1.tgz \
 --set global.accept_eula=true \
 --set global.storage_backend_type=nucleus \
 --set global.nucleus.server=<Omniverse Nucleus server hostname or IP> \
 --set global.nucleus.username=<Omniverse service account name> \
 --set global.nucleus.password=<Omniverse service account password> \
 --set global.secrets.create.auth=true \
 --set global.secrets.create.registry=true \
 --set global.ngcAPIKey=<NGC API KEY> \
 --set api-gateway.image.pullSecrets={nvcr.io} \
 --set deepsearch-crawler.crawler.extraConfig.exclude_patterns={"omniverse://[^\/]+/NVIDIA.*"}

NOTE: Please refer to NGC documentation for generating the NGC API Key.

NOTE: The /NVIDIA sample data mount is excluded from indexing by default. To enable its indexing, remove the corresponding line from the command. You can also customize URL patterns for indexing, as explained in the Indexing path filtering section.

Alternative installation options

The above command creates all required secrets on helm chart deployment, which is done for convenience. It may, however, be not acceptable in some deployment scenarios to pass credential information through helm chart values. To address this issue it is possible to create kubernetes secret with access information manually with the following command:

kubectl create secret generic deepsearch-service-account \
  --from-literal=username=<Omniverse service account name> \
  --from-literal=password=<Omniverse service account password>

Similarly, it is possible to create a registry access secret with the following command:

kubectl create secret docker-registry nvcr.io \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password=<NGC API KEY> \
  --docker-email=<your-email>

With the above secrets created, the helm chart installation command can be simplified to:

helm install <deployment name> usdsearch-1.1.1.tgz \
  --set global.accept_eula=true \
  --set global.storage_backend_type=nucleus \
  --set global.nucleus.server=<Omniverse Nucleus server hostname or IP> \
  --set deepsearch-crawler.crawler.extraConfig.exclude_patterns={"omniverse://[^\/]+/NVIDIA.*"}
  --set api-gateway.image.pullSecrets={nvcr.io} \
  --set global.imagePullSecrets={nvcr.io}

Nucleus API token

Instead of using service account it is possible to rely on Nucleus API token. To do so, you need to input $omni-api-token as the username and the API token value as the password. Additionally you need to pass the following parameter to disable permissions check that is not required for Nucleus API tokens:

  --set global.nucleus.assert_admin_user='false'

USD Search REST API Authentication

By default, when connected to an Omniverse Nucleus server, search service verifies that the user has access to retrieved assets before returning them. Therefore, in order to access search functionality, it is required to provide with one of the following authentication methods:

• service account username / password pair

  • alternatively is it possible to rely on Nucleus API token and provide $omni-api-token as the username and the API token value as the password.

• Nucleus connection token that is generated by the client library.

Admin Access Key

At service deploy time it is possible to configure admin access key that allows by-passing this access verification step by setting the access_key configuration parameter as follows:

    --set ngsearch.microservices.search_rest_api.admin_authentication.access_key=<some key value>

when this field is left unset the access key will be auto-generated and stored in a configmap on the kubernetes cluster. To retrieve the value of this key you can run the following:

export NAMESPACE=<deployment namespace>
export HELM_NAME=<deployment name>
echo $(kubectl get cm $HELM_NAME-ngsearch-env-config -n $NAMESPACE -o "jsonpath={.data.DEEPSEARCH_BACKEND_ADMIN_ACCESS_KEY}")

Alternatively it is possible to run the following command:

helm status <deployment name>

which will show some information about the deployment and the list of useful commands. The value of Admin access key will be printed there as part of Accessing USD Search section.

Disable Access verification

If access verification is not desired it is possible to disable it by providing the following argument to the installation or upgrade command:

    --set ngsearch.microservices.search_rest_api.enable_access_verification=false

API endpoint access

All the USD Search functionality is unified under a single API gateway. API Gateway service configuration. By default the ClusterIP service type is used, however, it is possible to override this and make the endpoint publicly accessible. Please refer to NGINX helm chart service configuration for a complete list of settings.

Below you can find several sample configurations for the API gateway endpoint, depending on the desired service type, and instructions on how to access it:

• ClusterIP (default) - If the ClusterIP service type is used, you can access the API externally via port-forwarding from the Kubernetes cluster as described below. For internal access, use the service name directly.

  API Gateway port-forward example:

  kubectl port-forward -n <NAMESPACE> svc/<deployment name>-api-gateway 8080:80
  

  The endpoint should then be accessible at http://localhost:8080.

• NodePort - You can specify the NodePort service type to make the endpoint publicly accessible using the Kubernetes cluster’s external IP. Typically, NodePort values must be in the >30000 range. To enable this, use the following command line arguments:

      --set api-gateway.service.type=NodePort \
      --set api-gateway.service.nodePorts.http=<NodePort value>
  

  the service will then be accessible at http://<external cluster IP>:<NodePort value>

  NOTE: Please refer to Kubernetes documentation more information about NodePort type services.

• Ingress - You can enable Ingress to make the service available at a specified hostname. This requires an Ingress controller to be set up, and you may need to specify an Ingress class that matches your controller. Use the following command line arguments to configure it:

      --set api-gateway.ingress.enabled=true \
      --set api-gateway.ingress.hostname=<service hostname> \
      --set api-gateway.ingress.ingressClassName=<ingress class name, e.g. 'nginx'>
  

  the service will then be accessible at http://<service hostname>

Monitoring

The USD Search Helm chart includes monitoring functionality that sets up metrics collection and pre-configured dashboards for tracking background indexing progress and the overall system state. This requires the Prometheus Operator and Grafana with dashboard provisioning. You can use the Kube Prometheus Stack to meet these requirements. To enable monitoring, provide the following command line arguments:

    --set deployGrafanaDashboards=true \
    --set deployServiceMonitors=true

With the above flag enabled the metrics will be automatically scraped by Prometheus service (part of Prometheus stack) and visualized in Grafana as USD Search / Plugin Processing Dashboard and USD Search / Metadata Indexing and Crawler Dashboard dashboards.

Post-installation

Deployment status check

After the helm chart is installed you can check the state of the deployment as follows:

helm status <deployment name>

This command should print the following to the standard output:

NAME: <deployment name>
LAST DEPLOYED: <deployment time>
NAMESPACE: <deployment namespace>
STATUS: deployed
REVISION: 1
NOTES:
  ...

It will also show some useful commands for checking the following:

• General deployment information (e.g. deployment name and namespace)
• Storage backend information
• USD Search API service information
• USD Search API service access information
• Couple of useful commands to check pod status
• Clean-up commands configured specifically for the deployment

Testing

The first installation of deployment may take some time, as all the containers will be pulled from the registry. Subsequent installations, however, will be faster.

In order to verify that deployment is successful and the services are working as expected it is possible to run the following

helm test <deployment name>

For a successful deployment this should print the following to standard output:

...
TEST SUITE:     <deployment name>-ags-api-verification
Last Started:   Fri Nov  8 12:01:47 2024
Last Completed: Fri Nov  8 12:01:52 2024
Phase:          Succeeded
TEST SUITE:     <deployment name>-database-search-api-verification
Last Started:   Fri Nov  8 12:01:53 2024
Last Completed: Fri Nov  8 12:01:59 2024
Phase:          Succeeded
TEST SUITE:     <deployment name>-s3-storage-connection-verification
Last Started:   Fri Nov  8 12:01:59 2024
Last Completed: Fri Nov  8 12:02:10 2024
Phase:          Succeeded
...

In case some of these test return a Failed status, please refer to the following resources for debugging the deployment:

• Frequently asked questions section
• Kubernetes application debugging guide

Uninstall

To uninstall the USD Search API deployment, run the following command:

helm uninstall <deployment name>

Rendering jobs clean-up

Rendering jobs created during deployment are not automatically removed on helm uninstall and may occupy resources on the cluster. In order to remove them run the following command:

kubectl delete jobs \
    -l 'app.kubernetes.io/instance=<deployment name>,deepsearch.job-type=rendering' \
    --field-selector status.successful=0

This command only removes those jobs are running or failed execution. Successfully terminated jobs do not occupy any resources on the kubernetes cluster and would be automatically removed after 1 hour, so it is not necessary to clean them up. Nevertheless, in some cases it may be preferred to remove those jobs as well, which could be done with the following command:

kubectl delete jobs \
    -l 'app.kubernetes.io/instance=<deployment name>,deepsearch.job-type=rendering' \
    --field-selector status.successful=1

Secrets and persistent volumes clean-up

Secrets and persistent volume claims (PVCs) are not automatically removed on helm uninstall. They are retained in case the instance needs to be re-created. In some cases it could be desired to remove them as well (e.g. in case a clean install from scratch is desired). The following set of commands could be used to achieve this:

• Secrets:

kubectl delete secrets $(kubectl get secrets \
    -l 'app.kubernetes.io/instance=<deployment name>' \
    -o custom-columns=":metadata.name" | awk '{print}' ORS=' ')

• Persistent volume claims (PVCs):

kubectl delete pvc $(kubectl get pvc \
    -l 'app.kubernetes.io/instance=<deployment name>' \
    -o custom-columns=":metadata.name" | awk '{print}' ORS=' ')
kubectl delete pvc data-neo4j-0

Advanced configuration

NOTE: Changing the settings provided below is typically not required for a standard installation of USD Search. These settings, however, allow customizing the deployment and tuning it for the particular use-case and infrastructure.

Below several additional parameters that allow customizing USD Search service are described. For convenience, instead of providing them all as command line arguments, it is recommended to pass them to the installation command as a configuration file as follows:

helm install .... -f my-usdsearch-config.yaml

where my-usdsearch-config.yaml file can have the following additional settings.

Indexing path filtering

By default USD Search indexes the all the assets that can be found on the server. It is, however, possible to explicitly define which URL patterns should be included in indexing and which should be excluded. The URL definition supports python regex syntax. In order to include / exclude some of the file patterns - it is possible to specify the following in the my-usdsearch-config.yaml file:

deepsearch-crawler:
  crawler:
    extraConfig:
      include_patterns:
      - <.*regexp of the folder that needs to be included.*>
      exclude_patterns:
      - <.*regexp of the folder that needs to be excluded.*>

Plugins

USD Search API service is built as a collection of plugins. These plugins are designed to focus on specific task that are outlined below:

• asset_graph_generation - extracts USD prim structure and metadata from USD assets

  • supports USD asset formats (.usd, .usda, .usdc, .usdz)
  • extracts all dependencies of a USD stage and stores it as a graph in Neo4j database
  • extracts asset properties, please refer to USD properties search section for more details on how the functionality could be configured for different use-cases.
  • enabled by default, if not required - could be disabled by providing the following command line argument during helm installation:

      --set deepsearch.plugins.asset_graph_generation.active=False
    

• image_to_embedding - extracts CLIP embeddings from image data

  • supports all the image file formats supported by PIL library and .exr file format
  • enabled by default, if not required - could be disabled by providing the following command line argument during helm installation:

      --set deepsearch.plugins.image_to_embedding.active=False
    

• image_to_vision_metadata - uses VLM configured in VLM-based automatic captioning and tagging section to automatically generate captions and tags from images

  • supports all the image file formats supported by PIL library and .exr file format
  • requires VLM to be properly configured, please refer to VLM-based automatic captioning and tagging section for more information
  • disabled by default, if not required - could be disabled by providing the following command line argument during helm installation:

      --set deepsearch.plugins.image_to_vision_metadata.active=True
    

• rendering_to_embedding - renders USD assets from multiple views and extracts CLIP embeddings from these renderings

  • supports USD asset formats (.usd, .usda, .usdc, .usdz)
  • enabled by default, if not required - could be disabled by providing the following command line argument during helm installation:

      --set deepsearch.plugins.rendering_to_embedding.active=False
    

• rendering_to_vision_metadata - renders USD assets from multiple views and uses VLM configured in VLM-based automatic captioning and tagging section to automatically generate captions and tags from renderings of these assets

  • supports USD asset formats (.usd, .usda, .usdc, .usdz)
  • requires VLM to be properly configured, please refer to VLM-based automatic captioning and tagging section for more information
  • disabled by default, if not required - could be disabled by providing the following command line argument during helm installation:

      --set deepsearch.plugins.rendering_to_vision_metadata.active=True
    

• thumbnail_generation - renders USD asset and uploads rendered image to the storage backend to serve as asset thumbnail

  • supports USD asset formats (.usd, .usda, .usdc, .usdz)
  • enabled by default, if not required - could be disabled by providing the following command line argument during helm installation:

      --set deepsearch.plugins.thumbnail_generation.active=False
    

• thumbnail_to_embedding - extracts CLIP embeddings from thumbnails of assets

  • any asset type is supported, provided this asset has a thumbnail
  • currently thumbnails are expected to be in the Omniverse Nucleus format:
    • the thumbnail images should be found in the .thumbs/256x256/ folder next to the asset.
    • they should have image should have either <original asset name>.png or <original asset name>.auto.png name.
  • enabled by default, if not required - could be disabled by providing the following command line argument during helm installation:

      --set deepsearch.plugins.thumbnail_to_embedding.active=False
    

• thumbnail_to_vision_metadata - uses VLM configured in VLM-based automatic captioning and tagging section to automatically generate captions and tags from thumbnails of assets

  • any asset type is supported, provided this asset has a thumbnail
  • currently thumbnails are expected to be in the Omniverse Nucleus format:
    • the thumbnail images should be found in the .thumbs/256x256/ folder next to the asset.
    • they should have image should have either <original asset name>.png or <original asset name>.auto.png name.
  • requires VLM to be properly configured, please refer to VLM-based automatic captioning and tagging section for more information
  • disabled by default, if not required - could be disabled by providing the following command line argument during helm installation:

      --set deepsearch.plugins.thumbnail_to_vision_metadata.active=True
    

Horizontal Pod Autoscaler

Each of the plugins is deployed as a separate k8s deployment that could be horizontally scaled. By default this functionality is disabled to avoid occupying to many resources. In order to enable it, one could add the following to individual plugin configuration.

hpa:
  enabled: true
  minReplicas: <desired minimum number of replicas (default 1) >
  maxReplicas: <desired maximum number of replicas (default 1) >
  targetCPUUtilizationPercentage: <desired target CPU utilization (default 80) >

for example for image_to_embedding plugin the configuration can look like this:

deepsearch:
  # ...
  plugins:
    # ...
    image_to_embedding:
      hpa:
        enabled: true
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilizationPercentage: 80

Concurrent processing

Some plugins may rely on external services and do not do a lot of processing themselves. In order to increase throughput it is possible to increase concurrency for each plugin by setting n_concurrent_queue_workers parameter to the desired value. For those plugins that require USD Asset rendering this value is set to 256 by default, others have this parameter set to 1.

Adjusting the value of this parameter may increase throughput and can be done as in the following example:

deepsearch:
  # ...
  plugins:
    # ...
    thumbnail_to_embedding:
      n_concurrent_queue_workers: 8

Rendering Job configuration

For rendering USD asset USD Search API rely on rendering jobs that are scheduled by Kubernetes on demand. If needed - rendering Job configuration could be adjusted.

Limit the number of Kit rendering instances

USD Search relies on kubernetes native scheduling functionality for scheduling rendering jobs depending on the availability of the resources on the kubernetes cluster. In some cases, however, (e.g. when working with Omniverse Nucleus server backend) it could be desired to limit the number of parallel rendering jobs created in order to control the load that is imposed on the storage backend.

To do so, one could use the maxRenderingJobsCount parameter, which could be provided to the helm installation command via the following command line argument:

 --set deepsearch.microservices.k8s_renderer.maxRenderingJobsCount=<max number of parallel rendering jobs>

or by modifying the my-usdsearch-config.yaml file as follows:

deepsearch:
  microservices:
    k8s_renderer:
      maxRenderingJobsCount: <max number of parallel rendering jobs>

If this parameter is left unset - no limit on the number of rendering jobs will be applied.

Resources

Resource requests and limits for each individual rendering job could be adjusted by updating the resources parameter in the deepsearch.microservices.k8s_renderer section in my-usdsearch-config.yaml file. The default settings for resource requests and limits for the job are outlined below:

requests:
    memory: "30Gi"
    cpu: "4"
    nvidia.com/gpu: "1"
limits:
    memory: "30Gi"
    cpu: "11"
    nvidia.com/gpu: "1"

Persistence (experimental)

Rendering job is using Omniverse Kit to render USD assets. When doing rendering Omniverse Kit saves some information in cache to make processing more efficient. This information includes shader cache, which takes between 100 and 300 seconds to compute. By default cache it created as a memory volume that is preserved during the lifetime of the job and is removed after it has completed.

In order to optimize processing it is, however, possible to persist this cached information, so that subsequent runs of the job are faster. In order to achieve this it is required to appropriately configure deepsearch.persistence section.

For each of these cache locations there are several set-up options that are controlled by setting the type of cache:

• emptyDir: creates the cache volume that is available only through out the lifetime of the job. It does not keep cache information during between the runs, which reduces performance as every time shaders need to be re-complied and shader cache re-created.
• pvc: creates PVC to for caching the data, which allows it to be preserved between execution, which in turn improves the speed of sub-sequent rendering runs. When using pvc persistence type it is possible to provide a custom PVC name by setting persistentVolumeClaim.claimName accordingly. Alternatively, it is possible to set pvc.createPvc to true in which case PVC will be automatically created on demand.
  • NOTE: In case of a custom PVC setup - it is important to make sure it is set with accessModes: [ReadWriteMany] so that multiple workers are able to access it. Not all storage classes support this setting (e.g. in AWS EFS storage class is required), so it is important to make sure that an appropriate storage class is used.
• hostPath: uses local host storage to store cache information
  • NOTE: when using local storage, the target directory hostPath.path has to be manually created on the host machine.

NOTE: This is an experimental feature. Some functionality may not be supported in case of the heterogenous GPU setup.

USD properties search

Asset Graph Search (AGS) component allows searching various properties defined as part of USD files. By default all the properties that are have the semantic: prefix are indexed, however it is possible to customize this behavior as follows:

plugins:
  asset_graph_generation:
    indexed_property_prefixes: "<custom_prefix_1>:,<custom_prefix_2>:"

Here <custom_prefix_1>, <custom_prefix_2> are the prefixes of USD properties that should be indexed. One could provide multiple prefixes separated by commas.

Alternatively, it is possible to specify the list of property names that should be indexed as follows:

plugins:
  asset_graph_generation:
    indexed_properties: "<property_name1>,<property_name2>,<property_name3>"

Here <property_name1>,<property_name2>,<property_name3> is the list of property names that should be indexed. One could provide multiple property names separated by commas. If indexed_properties parameter remains unset - all the properties with the prefixes defined above are indexed.

NOTE: Only the properties from the default prims inside a USD stage are indexed and searchable.

VLM-based automatic captioning and tagging

Vision endpoint allows to automatically tag and caption various assets stored on the storage backend using an external Vision Language Model (VLM) service.

The following types of VLM services providers could be configured with the helm chart:

• azure_openai
• openai
• nvidia_ai_endpoints
• anthropic
• mistralai

In order to select one VLM for the whole service globally you can pass the following command line parameter:

    --set deepsearch.vision_endpoint.vlm_service=<VLM service type>

or specify deepsearch.vision_endpoint.vlm_service setting in the my-usdsearch-config.yaml file as follows:

deepsearch:
  vision_endpoint:
    vlm_service: <VLM service type>

NOTE: This setting can be overwritten for any USD Search API plugin (e.g. rendering_to_vision_metadata plugin) by passing the following command line arguments:

    --set deepsearch.plugins.rendering_to_vision_metadata.vision_endpoint.vlm_service=<VLM service type>

Alternatively, for each plugin your can provide respective setting in the my-usdsearch-config.yaml file as follows:

deepsearch:
  plugins:
    rendering_to_vision_metadata:
      vision_endpoint:
        vlm_service: <VLM service type>

If no vision_endpoint setting is provided for the plugin, the global VLM configuration will be used.

For more information about the available USD Search API plugins, please refer to Plugin Settings section.

VLM services

Below you can find a list of VLM service providers with respective parameters that could be configured with USD Search API.

Anthropic

Anthropic VLM endpoint configuration.

It is required to provide a secret with the API key for accessing Anthropic service, which could be done with the following command:

kubectl create secret generic anthropic-vlm-api-key-secret \
    --from-literal=api-key=<Anthropic API Key>

Alternatively, if such secret is not created in advance it is possible to automatically by providing the following command line arguments during the first helm installation:

    --set global.secrets.create.vlm=true \
    --set deepsearch.vision_endpoint.anthropic.api_key=<Anthropic API Key>

If it possible to customize various parameters of the Anthropic VLM endpoint. This can be done using the following command line arguments:

    --set deepsearch.vision_endpoint.anthropic.parameters.<parameter name>=<parameter value>

The full list of parameters with pre-set default values can be found below:

max_tokens: 1024
model: claude-3-5-sonnet-latest
temperature: 0

Azure OpenAI

Azure OpenAI VLM endpoint configuration.

It is required to provide a secret with the API key for accessing Azure OpenAI service, which could be done with the following command:

kubectl create secret generic azure-openai-vlm-api-key-secret \
    --from-literal=api-key=<Azure OpenAI API Key>

Alternatively, if such secret is not created in advance it is possible to automatically by providing the following command line arguments during the first helm installation:

    --set global.secrets.create.vlm=true \
    --set deepsearch.vision_endpoint.azure_openai.api_key=<Azure OpenAI API Key>

If it possible to customize various parameters of the Azure OpenAI VLM endpoint. This can be done using the following command line arguments:

    --set deepsearch.vision_endpoint.azure_openai.parameters.<parameter name>=<parameter value>

The full list of parameters with pre-set default values can be found below:

api_version: 2023-08-01-preview
azure_deployment: gpt-4o-20240806
azure_endpoint: null
max_tokens: 1024
model: gpt-4o-20240806
temperature: 0

Mistral AI

Mistral AI VLM endpoint configuration.

It is required to provide a secret with the API key for accessing Mistral AI service, which could be done with the following command:

kubectl create secret generic mistralai-vlm-api-key-secret \
    --from-literal=api-key=<Mistral AI API Key>

Alternatively, if such secret is not created in advance it is possible to automatically by providing the following command line arguments during the first helm installation:

    --set global.secrets.create.vlm=true \
    --set deepsearch.vision_endpoint.mistralai.api_key=<Mistral AI API Key>

If it possible to customize various parameters of the Mistral AI VLM endpoint. This can be done using the following command line arguments:

    --set deepsearch.vision_endpoint.mistralai.parameters.<parameter name>=<parameter value>

The full list of parameters with pre-set default values can be found below:

max_tokens: 1024
model: pixtral-large-latest
temperature: 0

NVIDIA AI Endpoints

NVIDIA AI Endpoints VLM endpoint configuration.

It is required to provide a secret with the API key for accessing NVIDIA AI Endpoints service, which could be done with the following command:

kubectl create secret generic nvidia-ai-endpoints-vlm-api-key-secret \
    --from-literal=api-key=<NVIDIA API Key>

Alternatively, if such secret is not created in advance it is possible to automatically by providing the following command line arguments during the first helm installation:

    --set global.secrets.create.vlm=true \
    --set deepsearch.vision_endpoint.nvidia_ai_endpoints.api_key=<NVIDIA API Key>

If it possible to customize various parameters of the NVIDIA AI Endpoints VLM endpoint. This can be done using the following command line arguments:

    --set deepsearch.vision_endpoint.nvidia_ai_endpoints.parameters.<parameter name>=<parameter value>

The full list of parameters with pre-set default values can be found below:

max_tokens: 1024
model: nvidia/neva-22b
temperature: 0.01

OpenAI

OpenAI VLM endpoint configuration.

It is required to provide a secret with the API key for accessing OpenAI service, which could be done with the following command:

kubectl create secret generic openai-vlm-api-key-secret \
    --from-literal=api-key=<OpenAI API Key>

Alternatively, if such secret is not created in advance it is possible to automatically by providing the following command line arguments during the first helm installation:

    --set global.secrets.create.vlm=true \
    --set deepsearch.vision_endpoint.openai.api_key=<OpenAI API Key>

NOTE: Instead of relying on the OpenAI model, it is possible to provide on a custom VLM model endpoint, provided this custom VLM model has OpenAI API compatible interface. This could be achieved by appropriately setting the base_url parameter:

    --set deepsearch.vision_endpoint.openai.parameters.base_url=<your custom LLM model>

If it also possible to customize various other parameters of the OpenAI VLM endpoint. This can be done using the following command line arguments:

    --set deepsearch.vision_endpoint.openai.parameters.<parameter name>=<parameter value>

The full list of parameters with pre-set default values can be found below:

base_url: null
max_tokens: 1024
model: gpt-4o
temperature: 0

Customization

It is possible to customize the type of information extracted by the VLM-based auto-captioning system. There are two ways how this can be achieved:

1. By providing a custom image prompt.
2. By providing a custom list of metadata fields.

Each of the above is described in the following sections.

Image prompt configuration

Image prompt provided to the VLM-based auto-captioning system is fully customizable and can be controlled by setting deepsearch.vision_endpoint.image_prompt parameter in the values.yaml file. Below is the default configuration for the image prompt, however it could be adjusted to better fit the needs of the specific use case.

NOTE: there are two fields defined in the prompt: {metadata_keys} and {metadata_fields}. Those get populated with the respective values defined in deepsearch.vision_endpoint.metadata_fields parameter.

Write a detailed analysis of a set of image thumbnails that display different angles of a 3D object.
Your analysis should focus on identifying key features, textures, colors, and any other notable characteristics visible from the images.

Based on your analysis, generate a JSON object that encapsulates the metadata about the 3D object.
This metadata should include searchable terms and keywords that accurately describe the object's visual aspects, such as shape, color, texture, and any unique features.
Additionally, craft a concise caption that summarizes the object's appearance and distinctive attributes.

Ensure that the JSON object is structured in a way that facilitates its inclusion in a searchable index, making the 3D object easily discoverable based on its visual characteristics.
The metadata should be comprehensive yet succinct, enabling efficient search and retrieval in a database or search engine context.

The JSON output should follow this structure:
{metadata_keys}

Instructions:
{metadata_fields}

Your analysis must strive to be as precise and comprehensive as possible, using only the visual information available in the provided thumbnails.
Assume the role of an intelligent vision system tasked with generating actionable metadata for cataloging or identification purposes.

Metadata fields configuration

Metadata fields extracted by the VLM-based auto-captioning system could be adjusted to the target use-case by appropriately setting the deepsearch.vision_endpoint.metadata_fields parameter in the values.yaml file. This parameter is a list fields each of which is required to to have the following settings:

• name: name of the metadata field
• description: description of the metadata field and some guidance for the VLM model on how this metadata field should be extracted.
• type: type of the metadata field

The following represents the default configuration for the metadata fields:

- description: general category or type of the object. For example, it could be truck, airplane, toy, plant, etc. - all values should be returned comma separated.
  name: object_type
  type: string
- description: context or setting in which the object is typically found. This could be warehouse, living room, office, etc. - all values should be returned comma separated.
  name: scene_type
  type: string
- description: colors of the object. Take into account both the base and accent colors. - all values should be returned comma separated.
  name: colors
  type: string
- description: appearance and texture visible in the thumbnails, speculate on the materials the object is made from. - all values should be returned comma separated.
  name: materials
  type: string
- description: short, descriptive text about the object, focusing on its unique or distinguishing features.
  name: caption
  type: string
- description: searchable terms and keywords derived from the object_type, scene_type, colors, materials, and caption. These should encapsulate the essence of the 3D object, making it easily searchable and identifiable. - all values should be returned comma separated.
  name: tags
  type: string

Values

For convenience, configurable values of this Helm Chart are outlined in the sections below.

Global settings

false

{
  "ags": {
    "annotations": {
      "helm.sh/hook": "test",
      "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded",
      "helm.sh/hook-weight": "0"
    },
    "backoff_limit": 0,
    "enabled": true
  },
  "rest_api": {
    "annotations": {
      "helm.sh/hook": "test",
      "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded",
      "helm.sh/hook-weight": "0"
    },
    "backoff_limit": 0,
    "enabled": true
  },
  "storage_backend": {
    "annotations": {
      "helm.sh/hook": "pre-install,test",
      "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded",
      "helm.sh/hook-weight": "0"
    },
    "backoff_limit": 0,
    "enabled": true
  }
}

searches: []

enabled: true
type: "triton_server"
endpoint: ""

false

null

null

"ngc-api"

"nvcr.io"

""

"nvcr.io/nvidia/usdsearch"

{
  "annotations": {
    "helm.sh/resource-policy": "keep"
  },
  "create": {
    "auth": false,
    "ngc_api": false,
    "registry": false,
    "vlm": false
  }
}

"s3"

OTEL_SDK_DISABLED: "true"
OTEL_EXPORTER_OTLP_ENDPOINT: "http://tempo:4318"

Embedding service settings

The following parameters describe embedding service configuration settings:

1

requests:
    nvidia.com/gpu: 1
    cpu: 2
    memory: 7Gi
limits:
    nvidia.com/gpu: 1
    cpu: 4
    memory: 15Gi

emptyDir:
    medium: Memory
    sizeLimit: 256Mi

{}

Asset Graph Service additional settings

The following parameters allow configuring the Asset Graph Search (AGS) deployment:

5

""

true

Other settings

true

true

requests:
    cpu: 100m
    memory: 128Mi

{
  "replicas": 1
}

replicas: 1

enabled: true

{
  "replicas": 1
}

64

true

false

false

{
  "replicas": 1
}

{
  "replicas": 1
}

{
  "replicas": 1
}

Redis instance settings

Configuration for the default Redis instance, which gets deployed when redis_deployment.enabled is set to true.

"standalone"

enabled: False

redis.commonConfiguration: |
  appendonly yes
  save ""
  databases 32

disableCommands: []
persistence:
    enabled: True
    size: 64Gi
resources:
    limits:
        memory: 10Gi
        ephemeral-storage: 10Gi
        cpu: 1000m

replicaCount: 0

enabled: true

OpenSearch instance settings

Configuration for the default OpenSearch cluster, which gets deployed when opensearch_deployment.enabled is set to true.

"server.name: dashboards\nserver.host: \"0.0.0.0\"\nserver.ssl.enabled: false\nopensearch.ssl.verificationMode: none\nopensearch.username: kibanaserver\nopensearch.password: kibanaserver\nopensearch.requestHeadersWhitelist: [authorization, securitytenant]\nopensearch.hosts: [\"http://deepsearch-opensearch-cluster-master:9200\"]\n"

"DISABLE_SECURITY_DASHBOARDS_PLUGIN"

"true"

{}

false

""

""

""

"/"

"nginx"

{}

"http://deepsearch-opensearch-cluster-master:9200"

"deepsearch-opensearch-cluster"

opensearch.config."opensearch.yml": |
  network.host: 0.0.0.0
  #knn.algo_param.index_thread_qty: 8
  plugins:
    security:
      disabled: true
 

"DISABLE_INSTALL_DEMO_CONFIG"

"true"

"deepsearch-opensearch-cluster-master"

"-Xmx2048M -Xms2048M"

enabled: true
size: 100Gi

3

requests:
    cpu: "100m"
    memory: "4Gi"

enabled: false

{
  "enabled": false
}

false

enabled: true

Neo4j instance settings

Configuration for the default Neo4j instance, which gets deployed when neo4j_deployment.enabled is set to true.

server.config.strict_validation.enabled: "false"
server.memory.heap.initial_size: "8000m"
server.memory.heap.max_size: "8000m"

"[\"graph-data-science\", \"apoc\"]"

"neo4j"

name: neo4j
password: "password"
resources:
    requests:
        cpu: "4000m"
        memory: "14Gi"
    limits:
        cpu: "4000m"
        memory: "14Gi"

false

neo4j:
    enabled: true
    annotations: {}
    spec:
        type: ClusterIP

"ReadWriteOnce"

"100Gi"

"defaultStorageClass"

true

Maintainers

License

GOVERNING TERMS:

If you download the software and materials as available from the NVIDIA AI product portfolio, use is governed by the NVIDIA Software License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and the Product-Specific Terms for NVIDIA AI Products (found at https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/); except for the model which is governed by the NVIDIA AI Foundation Models Community License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-ai-foundation-models-community-license-agreement/.

If you download the software and materials as available from the NVIDIA Omniverse portfolio, use is governed by the NVIDIA Software License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and the Product-Specific Terms for NVIDIA Omniverse (found at NVIDIA Agreements | Enterprise Software | Product Specific Terms for Omniverse); except for the model which is governed by the NVIDIA AI Foundation Models Community License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-ai-foundation-models-community-license-agreement/.

FAQ

USD assets organization best practices

USD Search API service indexes individual assets that are stored on the storage backend. As described in Plugins section multiple asset formats are supported.

When it comes to USD assets, USD Search API service operates on the file level, so the whole USD asset / scene is indexed as a whole. If one has a large USD scene with multiple smaller prims baked in, only the overarching USD scene will be indexed as it would be the only actual file stored on the storage backend. Therefore, in case finding smaller objects that the scene is composed of is desired (e.g. in the case of doing in-scene search or finding scenes that contain specific objects), those individual objects should be:

• stored on the storage backend as individual USD assets,
• referenced in the main scene, instead of being baked into it.

Image-based search best practices

USD Search API relies on the NVCLIP model to extract embedding from images. NVCLIP model operates on RGB images of shape 224x224, therefore to achieve the best accuracy the input images should be squared and have at least 224x224 size.

If images have different dimensions - they will be rescaled such that the minimum of height and width is equal to 224 and then center-cropping will be applied to make sure aspect ratio is preserved and the NVCLIP model receives a squared input.

Redis Persistent Volume Claim

USD Search API relies on Redis for all internal caching and uses the official Redis Helm chart for installation, which itself relies on the Persistent Volume Claim mechanism. For the Redis installation provided with USD Search API, a Persistent Volume is required that would be then claimed by Redis. An example configuration is shown below for a Persistent Volume that uses storage on the local file system.

apiVersion: v1
kind: PersistentVolume
metadata:
name: sample-pv-name
spec:
accessModes:
- ReadWriteOnce
capacity:
  storage: 100Gi
local:
  path: /var/lib/omni/volumes/001
nodeAffinity:
  required:
    nodeSelectorTerms:
    - matchExpressions:
    - key: kubernetes.io/hostname
    operator: In
    values:
    - "node-name"
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem

For more information on different types of Persistent Volumes and their setup procedures, please refer to the official Kubernetes documentation.

Redis CrashLoopBackOff

In rare cases it could happen that Redis appendonly file ends up in a corrupted state. In this case the following line will be printed in the logs of the redis pod:

Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix ....

This may happen if the node that is running redis got unexpectedly terminated or has run out of space.

In order to fix this issue, please execute the following set of commands.

NOTE: The steps below assume that there is only a single instance of USD Search API installed in a provided namespace. If that is not the case the way REDIS_STATEFULSET_NAME and REDIS_POD_NAME are computed need to be updated.

# prepare some settings
export NAMESPACE=<namespace where USD Search API is running>
export REDIS_AOF_FILE_NAME=<corrupted file name from the Redis log>
# get the name of the statefulset that is controlling Redis
export REDIS_STATEFULSET_NAME=$(kubectl get statefulset -n $NAMESPACE -o custom-columns=":metadata.name" | grep redis)
# get the name of the pod running Redis
export REDIS_POD_NAME=$(kubectl get pods -n $NAMESPACE -o custom-columns=":metadata.name" | grep redis)
# patch Redis statefulset to sleep (to exit crashbackloop)
kubectl patch statefulset -n $NAMESPACE $REDIS_STATEFULSET_NAME -p '{"spec": {"template": {"spec":{"containers":[{"name": "redis","args": ["-c", "sleep 1000000000"]}]}}}}'
# give k8s 5 seconds to restart redis
sleep 5
# fix corrupted redis file
kubectl exec -it -n $NAMESPACE $REDIS_POD_NAME -- redis-check-aof --fix /data/appendonlydir/$REDIS_AOF_FILE_NAME
# revert statefulset patching
kubectl patch statefulset -n $NAMESPACE $REDIS_STATEFULSET_NAME -p '{"spec": {"template": {"spec":{"containers":[{"name": "redis","args": ["-c", "/opt/bitnami/scripts/start-scripts/start-master.sh"]}]}}}}'
# delete running container to make sure it restarts
kubectl delete pods $REDIS_POD_NAME

OpenSearch Persistent Volume Claim

When using the instance of OpenSearch provided with the USD Search API helm chart, the official OpenSearch Helm chart will be used for installation. This Helm chart by default installs a 3-Node OpenSearch instance and requires Persistent Volume storage. Creation of Persistent Volumes can be done in the exactly same way as described in the previous section.

Incorrect Service Registration Token with Omniverse Nucleus storage backend

When configuring USD Search API, a failure to register with the Nucleus Discovery service will happen if the provided Nucleus registration token is incorrect. If this occurs, the following error may be displayed:

Deployment: internal registration failed: DENIED

To solve this issue, the correct service registration token needs to be provided and can be located in the following subfolder within the Nucleus Docker Compose installation location:

base_stack/secrets/svc_reg_token

OpenSearch Virtual Memory (vm.max_map_count)

On some systems, the value of the kernel parameter vm.max_map_count may be too low for OpenSearch. If this is the case, it is required to update the default value for vm.max_map_count to at least 262144, as described in the OpenSearch installation documentation.

To check the current value, run this command:

cat /proc/sys/vm/max_map_count

To increase the value, add the following line to /etc/sysctl.conf:

vm.max_map_count=262144

Then run the following to reload and apply the settings change.

sudo sysctl -p

Storage backend connection

Helm chart installation assumes that storage backend (AWS S3 bucket or Omniverse Nucleus Server) is available before installation and valid credential information is provided.

For convenience we have included a helm pre-installation hook that checks the backend connection before installing of the helm chart.

If storage backend is not available, then depending on the backend type, one of the following errors will be printed during execution of helm install command:

Error: INSTALLATION FAILED: failed pre-install: 1 error occurred:
    * job test-nucleus-storage-connection-verification failed: BackoffLimitExceeded

or

Error: INSTALLATION FAILED: failed pre-install: 1 error occurred:
    * job test-s3-storage-connection-verification failed: BackoffLimitExceeded

It could happen that connection with the storage backend is broken after helm chart is installed. This could occur if the storage backend is unreachable for some reason. In this case, you may notice that many pods enter the CrashLoopBackOff state. To confirm that the issue is indeed related to the storage backend connection, you can do one of the following:

1. run helm test which will verify storage backend connection as follows:

   helm test <deployment name> --hide-notes
   

2. check the logs of any pod that entered CrashLoopBackOff and if you see ConnectionError messages - that would mean that storage backend is for some reason unavailable.

Search speed improvement

In case slow search speeds are encountered, it is possible to do several optimizations from the helm chart level.

Move embedding service to GPU

By default embedding service that is running NVCLIP model is relies on CPU for inference. It is however possible to make it use GPU, which significantly speeds up inference time. Please refer to Embedding service settings section.

Increase the number of OpenSearch replicas

If the cluster permits - it is possible to increase the number of OpenSearch replicas that is used. By default the helm chart is set to use 3 replicas, which we found to be sufficient in our experiments, however, this parameter could be overwritten. Therefore, it is recommended to check opensearch.replicas setting in my-usdsearch-config.yaml and adjust it according the amount of available resources. Alternatively it is possible to set the desired number of OpenSearch replicas as a command line argument as follows:

    --set opensearch.replicas=<desired number of OpenSearch replicas>

Indexing speed improvement

In case slow indexing speeds are encountered, it is possible to do several optimizations from the helm chart level.

Enable shader caching in Rendering jobs

By default Rendering Jobs are using memory medium for shader cache, which is only available during the lifetime of a job and therefore such cache needs to be re-calculated for each rendering job, which adds a significant overhead. Please refer to Rendering Job configuration section for more information on how to setup persistence.

Scale the cluster

Rendering jobs only get allocated, when enough resources are available on the cluster. So adding a node with more GPUs will linearly increase indexing speed.

Metrics are missing in Grafana with monitoring enabled

It could happen that Prometheus metrics do not appear in Grafana installed by Kube Prometheus Stack. The most common reason for such behavior is that Prometheus operator may not be configured to monitor all namespaces on the kubernetes cluster. In order to let Prometheus monitor all namespaces set the following in Kube Prometheus stack configuration:

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector: {}