usdsearch

USD Search API

Welcome to USD Search API Helm Chart!

Homepage: https://docs.omniverse.nvidia.com/services/latest/services/usd-search/overview.html

Requirements

Overview

This chart assumes that a Kubernetes cluster is already available and configured.

Depending on the ingress controller and namespace, some changes may be required.

Prerequisites

Kubernetes cluster with the following features enabled:

• Role-based access control (RBAC) - required for creating GPU-based asset rendering jobs
• NVIDIA k8s device plugin - required for execution of asset rendering jobs. Alternatively, NVIDIA GPU-operator (which includes NVIDIA k8s device plugin) could be used.

Optional:

• Metrics server - required for Horizontal Pod Autoscaling (HPA)
• Prometheus Stack - required for service monitoring
• Dynamic Persistent Volume provisioning - can be utilized by dependent helm charts (e.g. OpenSearch, Redis, Neo4j) in order to automatically provision required persistent volumes

Helm version is higher than 3.0.0. To check this run the following command

$ helm version
version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}

Packages and Access

Generate your NGC helm and container registry API Key prior to fetch helm chart. See onboarding guide here.

Fetch the latest helm chart from the registry

$ helm fetch https://helm.ngc.nvidia.com/nvidia/usdsearch/charts/usdsearch-1.1.0.tgz --username='$oauthtoken' --password=<YOUR API KEY>

Deployment

USD Search can be deployed to index data from an AWS S3 bucket or an Omniverse Nucleus server. Detailed deployment commands can be found in the sections below.

AWS S3

To deploy the USD Search Helm chart and connect it to an AWS S3 bucket for indexing, run the following command:

helm install <deployment name> usdsearch-1.1.0.tgz \
    --set global.accept_eula=true \
    --set global.storage_backend_type=s3 \
    --set global.s3.bucket_name=<S3 bucket name> \
    --set global.s3.region_name=<S3 bucket region>  \
    --set global.s3.aws_access_key_id=<AWS access key ID> \
    --set global.s3.aws_secret_access_key=<AWS secret access key>  \
    --set global.secrets.create.auth=true \
    --set global.secrets.create.registry=true \
    --set global.ngcAPIKey=<NGC API KEY> \
    --set api-gateway.image.pullSecrets={nvcr.io}

NOTE: Please refer to NGC documentation for generating the NGC API Key.

This command will create all required secrets on helm chart deployment, therefore subsequent installations / updates of the helm chart do not require providing credentials information and can be done as follows:

helm upgrade <deployment name> usdsearch-1.1.0.tgz --install \
    --set global.accept_eula=true \
    --set global.storage_backend_type=s3 \
    --set global.s3.bucket_name=<S3 bucket name> \
    --set global.s3.region_name=<S3 bucket region>  \
    --set api-gateway.image.pullSecrets={nvcr.io}

Additional parameters: re-scan frequency

USD Search re-scans the S3 bucket daily to detect new data while minimizing load. You can adjust the re-scan frequency by adding the following argument to the installation or upgrade command:

  --set global.s3.re_scan_timeout=<re scan timeout time in seconds>

NOTE: Setting this parameter too low may cause processing queues to grow faster than your USD Search instance can handle. It is recommended to monitor the processing queues in Grafana and adjust the parameter to prevent excessive queue growth.

S3proxy

USD Search supports integration with an S3-compatible proxy, such as S3proxy, for indexing. By default, no specific image is configured for S3proxy (image.repository.url is set to none), allowing flexibility for users to choose their preferred image. However, the official andrewgaul/s3proxy image is recommended, as it is Apache-licensed, regularly updated, and has been tested with USD Search to ensure compatibility.

To deploy the USD Search Helm chart with S3proxy, ensure that the s3proxy feature is enabled and configured in the values.yaml file or through Helm overrides. Below you will find commands to install S3proxy with different backends.

Examples for Different Storage Backends

Before deploying S3Proxy with different storage backends such as Azure Blob or Google Cloud Storage, create the necessary Kubernetes secret to store the jclouds identity and credential:

kubectl create secret generic s3proxy-credentials \
  --from-literal=jclouds-identity=<storage-backend-identity> \
  --from-literal=jclouds-credential="<storage-backend-key>" \
  --namespace <namespace>

Replace <storage-backend-identity> and <storage-backend-key> with the corresponding values from your storage backend account.

You can verify that the secret is correctly created with:

kubectl get secret s3proxy-credentials -n <namespace> -o yaml

Azure Blob Storage Backend

helm upgrade <deployment name> usdsearch-1.1.0.tgz --install \
  --namespace <namespace> \
  --set global.s3proxy.enabled=true \
  --set global.accept_eula=true \
  --set s3proxy.image.url=docker.io/andrewgaul/s3proxy:latest \
  --set global.s3.bucket_name=<your-bucket-name> \
  --set s3proxy.jclouds.provider="azureblob-sdk" \
  --set s3proxy.jclouds.endpoint="https://<azure-storage-account-name>.blob.core.windows.net/" \
  --set global.ngcAPIKey=<NGC-API-KEY> \
  --set global.secrets.create.registry=true \
  --set api-gateway.image.pullSecrets=<nvcr.io>

Google Cloud Storage Backend

helm upgrade <deployment name> usdsearch-1.1.0.tgz --install \
  --namespace <namespace> \
  --set global.s3proxy.enabled=true \
  --set global.accept_eula=true \
  --set global.s3.bucket_name=<your-gcs-bucket-name> \
  --set s3proxy.image.url=docker.io/andrewgaul/s3proxy:latest \
  --set s3proxy.jclouds.provider="google-cloud-storage" \
  --set s3proxy.jclouds.endpoint="https://storage.googleapis.com" \
  --set global.ngcAPIKey=<NGC-API-KEY> \
  --set global.secrets.create.registry=true \
  --set api-gateway.image.pullSecrets=<nvcr.io>

Documentation

For more details on configuring different storage backends, refer to the official S3proxy documentation or consult the cloud provider documentation for AWS S3, Azure Blob Storage, and Google Cloud Storage.

Omniverse Nucleus Server

USD Search service requires a service account with administrator rights in order to index the content stored on the Nucleus server. It is possible to use the main Nucleus service account, generated during the Nucleus service installation time. Alternatively, it is possible to create a dedicated service account for USD Search. For the exact steps on how such account could be created Please follow this guide.

When creating your own service account it is required to grant it admin access. This guide explains this process in detail:

After the service account's username and password are obtained, it is possible to deploy the USD Search Helm chart and connect it to an Omniverse Nucleus server for indexing, as follows:

helm install <deployment name> usdsearch-1.1.0.tgz \
    --set global.accept_eula=true \
    --set global.storage_backend_type=nucleus \
    --set global.nucleus.server=<Omniverse Nucleus server hostname or IP> \
    --set global.nucleus.username=<Omniverse service account name> \
    --set global.nucleus.password=<Omniverse service account password> \
    --set global.secrets.create.auth=true \
    --set global.secrets.create.registry=true \
    --set global.ngcAPIKey=<NGC API KEY> \
      --set api-gateway.image.pullSecrets={nvcr.io} \
    --set deepsearch-crawler.crawler.extraConfig.exclude_patterns={"omniverse://[^\/]+/NVIDIA.*"}

NOTE: Please refer to NGC documentation for generating the NGC API Key.

NOTE: The /NVIDIA sample data mount is excluded from indexing by default. To enable its indexing, remove the corresponding line from the command. You can also customize URL patterns for indexing, as explained in the Indexing path filtering section.

This command will create all required secrets on helm chart deployment, therefore subsequent installations / updates of the helm chart do not require providing credentials information and can be done as follows:

helm upgrade <deployment name> usdsearch-1.1.0.tgz --install \
    --set global.accept_eula=true \
    --set global.storage_backend_type=nucleus \
    --set global.nucleus.server=<Omniverse Nucleus server hostname or IP> \
    --set api-gateway.image.pullSecrets={nvcr.io} \
    --set deepsearch-crawler.crawler.extraConfig.exclude_patterns={"omniverse://[^\/]+/NVIDIA.*"}

Authentication

By default, when connected to an Omniverse Nucleus server, search service verifies that the user has access to retrieved assets before returning them. It is, therefore, required for the user to provide either username / password pair or a Nucleus authentication token in order to access search functionality.

Admin Access Key

At service deploy time it is possible to configure admin access key that allows by-passing this access verification step by setting the access_key configuration parameter as follows:

    --set ngsearch.microservices.search_rest_api.admin_authentication.access_key=<some key value>

when this field is left unset the access key will be auto-generated and stored in a configmap on the kubernetes cluster. To retrieve the value of this key you can run the following:

$ export NAMESPACE=<deployment namespace>
$ export HELM_NAME=<deployment name>
$ echo $(kubectl get cm $HELM_NAME-ngsearch-env-config -n $NAMESPACE -o "jsonpath={.data.DEEPSEARCH_BACKEND_ADMIN_ACCESS_KEY}")

Alternatively it is possible to run the following command:

helm status <deployment name>

which will show some information about the deployment and the list of useful commands. The value of Admin access key will be printed there as part of Accessing USD Search section.

Disable Access verification

If access verification is not desired it is possible to disable it by providing the following argument to the installation or upgrade command:

    --set ngsearch.microservices.search_rest_api.enable_access_verification=false

API endpoint access

All the USD Search functionality is unified under a single API gateway. API Gateway service configuration. By default the ClusterIP service type is used, however, it is possible to override this and make the endpoint publicly accessible. Please refer to NGINX helm chart service configuration for a complete list of settings.

Below you can find several sample configurations for the API gateway endpoint, depending on the desired service type, and instructions on how to access it:

• ClusterIP (default) - If the ClusterIP service type is used, you can access the API externally via port-forwarding from the Kubernetes cluster as described below. For internal access, use the service name directly.

  API Gateway port-forward example:

  $ kubectl port-forward -n <NAMESPACE> svc/<deployment name>-api-gateway 8080:80
  

  The endpoint should then be accessible at http://localhost:8080.

• NodePort - You can specify the NodePort service type to make the endpoint publicly accessible using the Kubernetes cluster’s external IP. Typically, NodePort values must be in the >30000 range. To enable this, use the following command line arguments:

      --set api-gateway.service.type=NodePort \
      --set api-gateway.service.nodePorts.http=<NodePort value>
  

  the service will then be accessible at http://<external cluster IP>:<NodePort value>

  NOTE: Please refer to Kubernetes documentation more information about NodePort type services.

• Ingress - You can enable Ingress to make the service available at a specified hostname. This requires an Ingress controller to be set up, and you may need to specify an Ingress class that matches your controller. Use the following command line arguments to configure it:

      --set api-gateway.ingress.enabled=true \
      --set api-gateway.ingress.hostname=<service hostname> \
      --set api-gateway.ingress.ingressClassName=<ingress class name, e.g. 'nginx'>
  

  the service will then be accessible at http://<service hostname>

Monitoring

The USD Search Helm chart includes monitoring functionality that sets up metrics collection and pre-configured dashboards for tracking background indexing progress and the overall system state. This requires the Prometheus Operator and Grafana with dashboard provisioning. You can use the Kube Prometheus Stack to meet these requirements. To enable monitoring, provide the following command line arguments:

    --set deployGrafanaDashboards=true \
    --set deployServiceMonitors=true

With the above flag enabled the metrics will be automatically scraped by Prometheus service (part of Prometheus stack) and visualized in Grafana as USD Search / Plugin Processing Dashboard and USD Search / Metadata Indexing and Crawler Dashboard dashboards.

Testing

The first installation of deployment may take some time, as all the containers will be pulled from the registry. Subsequent installations, however, will be faster.

In order to verify that deployment is successful and the services are working as expected it is possible to run the following

helm test <deployment name>

For a successful deployment this should print the following to standard output:

...
TEST SUITE:     <deployment name>-ags-api-verification
Last Started:   Fri Nov  8 12:01:47 2024
Last Completed: Fri Nov  8 12:01:52 2024
Phase:          Succeeded
TEST SUITE:     <deployment name>-database-search-api-verification
Last Started:   Fri Nov  8 12:01:53 2024
Last Completed: Fri Nov  8 12:01:59 2024
Phase:          Succeeded
TEST SUITE:     <deployment name>-s3-storage-connection-verification
Last Started:   Fri Nov  8 12:01:59 2024
Last Completed: Fri Nov  8 12:02:10 2024
Phase:          Succeeded
...

In case some of these test return a Failed status, please refer to the following resources for debugging the deployment:

• Frequently asked questions section
• Kubernetes application debugging guide

Advanced configuration

NOTE: Changing the settings provided below is typically not required for a standard installation of USD Search. These settings, however, allow customizing the deployment and tuning it for the particular use-case and infrastructure.

Below several additional parameters that allow customizing USD Search service are described. For convenience, instead of providing them all as command line arguments, it is recommended to pass them to the installation command as a configuration file as follows:

helm install .... -f my-usdsearch-config.yaml

where my-usdsearch-config.yaml file can have the following additional settings.

Indexing path filtering

By default USD Search indexes the all the assets that can be found on the server. It is, however, possible to explicitly define which URL patterns should be included in indexing and which should be excluded. The URL definition supports python regex syntax. In order to include / exclude some of the file patterns - it is possible to specify the following in the my-usdsearch-config.yaml file:

deepsearch-crawler:
  crawler:
    extraConfig:
      include_patterns:
      - <.*regexp of the folder that needs to be included.*>
      exclude_patterns:
      - <.*regexp of the folder that needs to be excluded.*>

Rendering Job configuration

For rendering USD asset USD Search API rely on rendering jobs that are scheduled by Kubernetes on demand. If needed - rendering Job configuration could be adjusted.

Resources

Resource requests and limits for each individual rendering job could be adjusted by updating the resources parameter in the deepsearch.microservices.k8s_renderer section in my-usdsearch-config.yaml file. The default settings for resource requests and limits for the job are outlined below:

requests:
    memory: "30Gi"
    cpu: "4"
    nvidia.com/gpu: "1"
limits:
    memory: "30Gi"
    cpu: "11"
    nvidia.com/gpu: "1"

Persistence (experimental)

Rendering job is using Omniverse Kit to render USD assets. When doing rendering Omniverse Kit saves some information in cache to make processing more efficient. This information includes shader cache, which takes between 100 and 300 seconds to compute. By default cache it created as a memory volume that is preserved during the lifetime of the job and is removed after it has completed.

In order to optimize processing it is, however, possible to persist this cached information, so that subsequent runs of the job are faster. In order to achieve this it is required to appropriately configure deepsearch.persistence section.

For each of these cache locations there are several set-up options that are controlled by setting the type of cache:

• emptyDir: creates the cache volume that is available only through out the lifetime of the job. It does not keep cache information during between the runs, which reduces performance as every time shaders need to be re-complied and shader cache re-created.
• pvc: creates PVC to for caching the data, which allows it to be preserved between execution, which in turn improves the speed of sub-sequent rendering runs. When using pvc persistence type it is possible to provide a custom PVC name by setting persistentVolumeClaim.claimName accordingly. Alternatively, it is possible to set pvc.createPvc to true in which case PVC will be automatically created on demand.
  • NOTE: In case of a custom PVC setup - it is important to make sure it is set with accessModes: [ReadWriteMany] so that multiple workers are able to access it. Not all storage classes support this setting (e.g. in AWS EFS storage class is required), so it is important to make sure that an appropriate storage class is used.
• hostPath: uses local host storage to store cache information
  • NOTE: when using local storage, the target directory hostPath.path has to be manually created on the host machine.

NOTE: This is an experimental feature. Some functionality may not be supported in case of the heterogenous GPU setup.

VLM-based automatic captioning and tagging

Vision endpoint allows to automatically tag and caption various assets stored on the storage backend using an external Vision Language Model (VLM) service.

The following types of VLM services providers could be configured with the helm chart:

• azure_openai
• openai
• nvidia_ai_endpoints
• anthropic
• mistralai

In order to select one VLM for the whole service globally you can pass the following command line parameter:

    --set deepsearch.vision_endpoint.vlm_service=<VLM service type>

or specify deepsearch.vision_endpoint.vlm_service setting in the my-usdsearch-config.yaml file as follows:

deepsearch:
  vision_endpoint:
    vlm_service: <VLM service type>

NOTE: This setting can be overwritten for any USD Search API plugin (e.g. rendering_to_vision_metadata plugin) by passing the following command line arguments:

    --set deepsearch.plugins.rendering_to_vision_metadata.vision_endpoint.vlm_service=<VLM service type>

Alternatively, for each plugin your can provide respective setting in the my-usdsearch-config.yaml file as follows:

deepsearch:
  plugins:
    rendering_to_vision_metadata:
      vision_endpoint:
        vlm_service: <VLM service type>

If no vision_endpoint setting is provided for the plugin, the global VLM configuration will be used.

For more information about the available USD Search API plugins, please refer to Plugin Settings section.

VLM services

Below you can find a list of VLM service providers with respective parameters that could be configured with USD Search API.

Anthropic

Anthropic VLM endpoint configuration.

It is required to provide a secret with the API key for accessing Anthropic service. If such secret is not created in advance it is possible to automatically by providing the following command line arguments during the first helm installation:

    --set global.secrets.create.vlm=true \
    --set deepsearch.vision_endpoint.anthropic.api_key=<NVIDIA API Key>

If it possible to customize various parameters of the Anthropic VLM endpoint. This can be done using the following command line arguments:

    --set deepsearch.vision_endpoint.anthropic.parameters.<parameter name>=<parameter value>

The full list of parameters with pre-set default values can be found below:

max_tokens: 1024
model: claude-3-5-sonnet-latest
temperature: 0

Azure OpenAI

Azure OpenAI VLM endpoint configuration.

It is required to provide a secret with the API key for accessing Azure OpenAI service. If such secret is not created in advance it is possible to automatically by providing the following command line arguments during the first helm installation:

    --set global.secrets.create.vlm=true \
    --set deepsearch.vision_endpoint.azure_openai.api_key=<Azure OpenAI API Key>

If it possible to customize various parameters of the Azure OpenAI VLM endpoint. This can be done using the following command line arguments:

    --set deepsearch.vision_endpoint.azure_openai.parameters.<parameter name>=<parameter value>

The full list of parameters with pre-set default values can be found below:

api_version: 2023-08-01-preview
azure_deployment: gpt-4o-20240806
azure_endpoint: null
max_tokens: 1024
model: gpt-4o-20240806
temperature: 0

Mistral AI

Mistral AI VLM endpoint configuration.

It is required to provide a secret with the API key for accessing Mistral AI service. If such secret is not created in advance it is possible to automatically by providing the following command line arguments during the first helm installation:

    --set global.secrets.create.vlm=true \
    --set deepsearch.vision_endpoint.mistralai.api_key=<NVIDIA API Key>

If it possible to customize various parameters of the Mistral AI VLM endpoint. This can be done using the following command line arguments:

    --set deepsearch.vision_endpoint.mistralai.parameters.<parameter name>=<parameter value>

The full list of parameters with pre-set default values can be found below:

max_tokens: 1024
model: pixtral-large-latest
temperature: 0

NVIDIA AI Endpoints

NVIDIA AI Endpoints VLM endpoint configuration.

It is required to provide a secret with the API key for accessing NVIDIA AI Endpoints service. If such secret is not created in advance it is possible to automatically by providing the following command line arguments during the first helm installation:

    --set global.secrets.create.vlm=true \
    --set deepsearch.vision_endpoint.nvidia_ai_endpoints.api_key=<NVIDIA API Key>

If it possible to customize various parameters of the NVIDIA AI Endpoints VLM endpoint. This can be done using the following command line arguments:

    --set deepsearch.vision_endpoint.nvidia_ai_endpoints.parameters.<parameter name>=<parameter value>

The full list of parameters with pre-set default values can be found below:

max_tokens: 1024
model: nvidia/neva-22b
temperature: 0.01

OpenAI

OpenAI VLM endpoint configuration.

It is required to provide a secret with the API key for accessing OpenAI service. If such secret is not created in advance it is possible to create it automatically by providing the following command line arguments during the first helm installation:

    --set global.secrets.create.vlm=true \
    --set deepsearch.vision_endpoint.openai.api_key=<OpenAI API Key>

NOTE: Instead of relying on the OpenAI model, it is possible to provide on a custom VLM model endpoint, provided this custom VLM model has OpenAI API compatible interface. This could be achieved by appropriately setting the base_url parameter:

    --set deepsearch.vision_endpoint.openai.parameters.base_url=<your custom LLM model>

If it also possible to customize various other parameters of the OpenAI VLM endpoint. This can be done using the following command line arguments:

    --set deepsearch.vision_endpoint.openai.parameters.<parameter name>=<parameter value>

The full list of parameters with pre-set default values can be found below:

base_url: null
max_tokens: 1024
model: gpt-4o
temperature: 0

Values

For convenience, configurable values of this Helm Chart are outlined in the sections below.

Global settings

false

{
  "ags": {
    "annotations": {
      "helm.sh/hook": "test",
      "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded",
      "helm.sh/hook-weight": "0"
    },
    "backoff_limit": 0,
    "enabled": true
  },
  "rest_api": {
    "annotations": {
      "helm.sh/hook": "test",
      "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded",
      "helm.sh/hook-weight": "0"
    },
    "backoff_limit": 0,
    "enabled": true
  },
  "storage_backend": {
    "annotations": {
      "helm.sh/hook": "pre-install,test",
      "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded",
      "helm.sh/hook-weight": "0"
    },
    "backoff_limit": 0,
    "enabled": true
  }
}

searches: []

enabled: true
type: "triton_server"
endpoint: ""

false

null

null

"ngc-api"

"nvcr.io"

""

"nvcr.io/nvidia/usdsearch"

{
  "annotations": {
    "helm.sh/resource-policy": "keep"
  },
  "create": {
    "auth": false,
    "ngc_api": false,
    "registry": false,
    "vlm": false
  }
}

"s3"

OTEL_SDK_DISABLED: "true"
OTEL_EXPORTER_OTLP_ENDPOINT: "http://tempo:4318"

Rendering Job settings

The following parameters describe Omniverse Kit-based rendering Job configuration

3600

false

map[]

requests:
    memory: "30Gi"
    cpu: "4"
    nvidia.com/gpu: "1"
limits:
    memory: "30Gi"
    cpu: "11"
    nvidia.com/gpu: "1"

3600

Plugin settings

USD Search API service is built as a collection of plugins, each of which does a specific job. These plugins can be optionally activated and deactivated by modifying the following configuration and setting the active parameter of specific plugins to either True or False.

Horizontal Pod Autoscaler

Each of the plugins is deployed as a separate k8s deployment that could be horizontally scaled. By default this functionality is disabled to avoid occupying to many resources. In order to enable it, one could add the following to individual plugin configuration.

hpa:
  enabled: true
  minReplicas: <desired minimum number of replicas (default 1) >
  maxReplicas: <desired maximum number of replicas (default 1) >
  targetCPUUtilizationPercentage: <desired target CPU utilization (default 80) >

for example for image_to_embedding plugin the configuration can look like this:

deepsearch:
  # ...
  plugins:
    # ...
    image_to_embedding:
      hpa:
        enabled: true
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilizationPercentage: 80

Concurrent processing

Some plugins may rely on external services and do not do a lot of processing themselves. In order to increase throughput it is possible to increase concurrency for each plugin by setting n_concurrent_queue_workers parameter to the desired value. For those plugins that require USD Asset rendering this value is set to 256 by default, others have this parameter set to 1.

Adjusting the value of this parameter may increase throughput and can be done as in the following example:

deepsearch:
  # ...
  plugins:
    # ...
    thumbnail_to_embedding:
      n_concurrent_queue_workers: 8

The full list of plugins with descriptions can be found below:

active: True

plugins:
  asset_graph_generation:
    active: true
    indexed_properties: "prop1,prop2,prop3"
    indexed_property_prefixes: "semantic:,custom:"

active: True

active: False

active: True
n_concurrent_queue_workers: 256

active: False
n_concurrent_queue_workers: 256

active: True
n_concurrent_queue_workers: 256

active: True

Embedding service settings

The following parameters describe embedding service configuration settings:

1

requests:
    nvidia.com/gpu: 1
    cpu: 2
    memory: 7Gi
limits:
    nvidia.com/gpu: 1
    cpu: 4
    memory: 15Gi

emptyDir:
    medium: Memory
    sizeLimit: 256Mi

{}

Asset Graph Service additional settings

The following parameters allow configuring the Asset Graph Search (AGS) deployment:

{
  "n_workers": 5
}

5

""

true

Other settings

true

true

requests:
    cpu: 100m
    memory: 128Mi

{
  "replicas": 1
}

replicas: 1

enabled: true

{
  "replicas": 1
}

64

true

false

false

{
  "replicas": 1
}

{
  "replicas": 1
}

{
  "replicas": 1
}

Redis instance settings

Configuration for the default Redis instance, which gets deployed when redis_deployment.enabled is set to true.

"standalone"

enabled: False

redis.commonConfiguration: |
  appendonly yes
  save ""
  databases 32

disableCommands: []
persistence:
    enabled: True
    size: 64Gi
resources:
    limits:
        memory: 10Gi
        ephemeral-storage: 10Gi
        cpu: 1000m

replicaCount: 0

enabled: true

OpenSearch instance settings

Configuration for the default OpenSearch cluster, which gets deployed when opensearch_deployment.enabled is set to true.

"server.name: dashboards\nserver.host: \"0.0.0.0\"\nserver.ssl.enabled: false\nopensearch.ssl.verificationMode: none\nopensearch.username: kibanaserver\nopensearch.password: kibanaserver\nopensearch.requestHeadersWhitelist: [authorization, securitytenant]\nopensearch.hosts: [\"http://deepsearch-opensearch-cluster-master:9200\"]\n"

"DISABLE_SECURITY_DASHBOARDS_PLUGIN"

"true"

{}

false

""

""

""

"/"

"nginx"

{}

"http://deepsearch-opensearch-cluster-master:9200"

"deepsearch-opensearch-cluster"

opensearch.config."opensearch.yml": |
  network.host: 0.0.0.0
  #knn.algo_param.index_thread_qty: 8
  plugins:
    security:
      disabled: true
 

"DISABLE_INSTALL_DEMO_CONFIG"

"true"

"deepsearch-opensearch-cluster-master"

"-Xmx2048M -Xms2048M"

enabled: true
size: 100Gi

3

requests:
    cpu: "100m"
    memory: "4Gi"

enabled: false

{
  "enabled": false
}

false

enabled: true

Neo4j instance settings

Configuration for the default Neo4j instance, which gets deployed when neo4j_deployment.enabled is set to true.

server.config.strict_validation.enabled: "false"
server.memory.heap.initial_size: "8000m"
server.memory.heap.max_size: "8000m"

"[\"graph-data-science\", \"apoc\"]"

"neo4j"

name: neo4j
password: "password"
resources:
    requests:
        cpu: "4000m"
        memory: "14Gi"
    limits:
        cpu: "4000m"
        memory: "14Gi"

false

neo4j:
    enabled: true
    annotations: {}
    spec:
        type: ClusterIP

"ReadWriteOnce"

"100Gi"

"defaultStorageClass"

true

Maintainers

License

GOVERNING TERMS:

If you download the software and materials as available from the NVIDIA AI product portfolio, use is governed by the NVIDIA Software License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and the Product-Specific Terms for NVIDIA AI Products (found at https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/); except for the model which is governed by the NVIDIA AI Foundation Models Community License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-ai-foundation-models-community-license-agreement/.

If you download the software and materials as available from the NVIDIA Omniverse portfolio, use is governed by the NVIDIA Software License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and the Product-Specific Terms for NVIDIA Omniverse (found at NVIDIA Agreements | Enterprise Software | Product Specific Terms for Omniverse); except for the model which is governed by the NVIDIA AI Foundation Models Community License Agreement (found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-ai-foundation-models-community-license-agreement/.

FAQ

Redis Persistent Volume Claim

USD Search API relies on Redis for all internal caching and uses the official Redis Helm chart for installation, which itself relies on the Persistent Volume Claim mechanism. For the Redis installation provided with USD Search API, a Persistent Volume is required that would be then claimed by Redis. An example configuration is shown below for a Persistent Volume that uses storage on the local file system.

apiVersion: v1
kind: PersistentVolume
metadata:
name: sample-pv-name
spec:
accessModes:
- ReadWriteOnce
capacity:
  storage: 100Gi
local:
  path: /var/lib/omni/volumes/001
nodeAffinity:
  required:
    nodeSelectorTerms:
    - matchExpressions:
    - key: kubernetes.io/hostname
    operator: In
    values:
    - "node-name"
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem

For more information on different types of Persistent Volumes and their setup procedures, please refer to the official Kubernetes documentation.

Redis CrashLoopBackOff

In rare cases it could happen that Redis appendonly file ends up in a corrupted state. In this case the following line will be printed in the logs of the redis pod:

Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix ....

This may happen if the node that is running redis got unexpectedly terminated or has run out of space.

In order to fix this issue, please execute the following set of commands.

NOTE: The steps below assume that there is only a single instance of USD Search API installed in a provided namespace. If that is not the case the way REDIS_STATEFULSET_NAME and REDIS_POD_NAME are computed need to be updated.

# prepare some settings
export NAMESPACE=<namespace where USD Search API is running>
export REDIS_AOF_FILE_NAME=<corrupted file name from the Redis log>
# get the name of the statefulset that is controlling Redis
export REDIS_STATEFULSET_NAME=$(kubectl get statefulset -n $NAMESPACE -o custom-columns=":metadata.name" | grep redis)
# get the name of the pod running Redis
export REDIS_POD_NAME=$(kubectl get pods -n $NAMESPACE -o custom-columns=":metadata.name" | grep redis)
# patch Redis statefulset to sleep (to exit crashbackloop)
kubectl patch statefulset -n $NAMESPACE $REDIS_STATEFULSET_NAME -p '{"spec": {"template": {"spec":{"containers":[{"name": "redis","args": ["-c", "sleep 1000000000"]}]}}}}'
# give k8s 5 seconds to restart redis
sleep 5
# fix corrupted redis file
kubectl exec -it -n $NAMESPACE $REDIS_POD_NAME -- redis-check-aof --fix /data/appendonlydir/$REDIS_AOF_FILE_NAME
# revert statefulset patching
kubectl patch statefulset -n $NAMESPACE $REDIS_STATEFULSET_NAME -p '{"spec": {"template": {"spec":{"containers":[{"name": "redis","args": ["-c", "/opt/bitnami/scripts/start-scripts/start-master.sh"]}]}}}}'
# delete running container to make sure it restarts
kubectl delete pods $REDIS_POD_NAME

OpenSearch Persistent Volume Claim

When using the instance of OpenSearch provided with the USD Search API helm chart, the official OpenSearch Helm chart will be used for installation. This Helm chart by default installs a 3-Node OpenSearch instance and requires Persistent Volume storage. Creation of Persistent Volumes can be done in the exactly same way as described in the previous section.

Incorrect Service Registration Token with Omniverse Nucleus storage backend

When configuring USD Search API, a failure to register with the Nucleus Discovery service will happen if the provided Nucleus registration token is incorrect. If this occurs, the following error may be displayed:

Deployment: internal registration failed: DENIED

To solve this issue, the correct service registration token needs to be provided and can be located in the following subfolder within the Nucleus Docker Compose installation location:

base_stack/secrets/svc_reg_token

OpenSearch Virtual Memory (vm.max_map_count)

On some systems, the value of the kernel parameter vm.max_map_count may be too low for OpenSearch. If this is the case, it is required to update the default value for vm.max_map_count to at least 262144, as described in the OpenSearch installation documentation.

To check the current value, run this command:

cat /proc/sys/vm/max_map_count

To increase the value, add the following line to /etc/sysctl.conf:

vm.max_map_count=262144

Then run the following to reload and apply the settings change.

sudo sysctl -p

Storage backend connection

Helm chart installation assumes that storage backend (AWS S3 bucket or Omniverse Nucleus Server) is available before installation and valid credential information is provided.

For convenience we have included a helm pre-installation hook that checks the backend connection before installing of the helm chart.

If storage backend is not available, then depending on the backend type, one of the following errors will be printed during execution of helm install command:

Error: INSTALLATION FAILED: failed pre-install: 1 error occurred:
    * job test-nucleus-storage-connection-verification failed: BackoffLimitExceeded

or

Error: INSTALLATION FAILED: failed pre-install: 1 error occurred:
    * job test-s3-storage-connection-verification failed: BackoffLimitExceeded

It could happen that connection with the storage backend is broken after helm chart is installed. This could occur if the storage backend is unreachable for some reason. In this case, you may notice that many pods enter the CrashLoopBackOff state. To confirm that the issue is indeed related to the storage backend connection, you can do one of the following:

1. run helm test which will verify storage backend connection as follows:

   helm test <deployment name> --hide-notes
   

2. check the logs of any pod that entered CrashLoopBackOff and if you see ConnectionError messages - that would mean that storage backend is for some reason unavailable.

Search speed improvement

In case slow search speeds are encountered, it is possible to do several optimizations from the helm chart level.

Move embedding service to GPU

By default embedding service that is running NVCLIP model is relies on CPU for inference. It is however possible to make it use GPU, which significantly speeds up inference time. Please refer to Embedding service configuration section.

Increase the number of OpenSearch replicas

If the cluster permits - it is possible to increase the number of OpenSearch replicas that is used. By default the helm chart is set to use 3 replicas, which we found to be sufficient in our experiments, however, this parameter could be overwritten. Therefore, it is recommended to check opensearch.replicas setting in my-usdsearch-config.yaml and adjust it according the amount of available resources. Alternatively it is possible to set the desired number of OpenSearch replicas as a command line argument as follows:

    --set opensearch.replicas=<desired number of OpenSearc replicas>

Indexing speed improvement

In case slow indexing speeds are encountered, it is possible to do several optimizations from the helm chart level.

Enable shader caching in Rendering jobs

By default Rendering Jobs are using memory medium for shader cache, which is only available during the lifetime of a job and therefore such cache needs to be re-calculated for each rendering job, which adds a significant overhead. Please refer to Rendering Job configuration section for more information on how to setup persistence.

Scale the cluster

Rendering jobs only get allocated, when enough resources are available on the cluster. So adding a node with more GPUs will linearly increase indexing speed.