Nemo Retriever Embedding Helm Chart

This Helm Chart simplifies Embedding NIM deployment on Kubernetes. It aims to support deployment with a variety of possible cluster, GPU and storage confurations.

NIMs are intended to be run on a system with NVIDIA GPUs, with the type and number of GPUs depending on the model. To use helm, you must have a Kubernetes cluster with appropriate GPU nodes and the GPU Operator installed.

Setting up the environment

If you haven't set up your NGC API key and do not know exactly which NIM you want to download and deploy, see the information in the User Guide.

This helm chart requires that you have a secret with your NGC API key configured for downloading private images, and one with your NGC API key (below named ngc-api). These will likely have the same key in it, but they will have different formats (dockerconfig.json vs opaque). See Creating Secrets below.

These instructions will assume that you have your NGC_API_KEY exported in the environment.

export NGC_API_KEY="<YOUR NGC API KEY>"

Fetching the Helm Chart

You can fetch the helm chart from NGC by executing the following command:

helm fetch https://helm.ngc.nvidia.com/nim/nvidia/charts/text-embedding-nim-1.0.0.tgz --username='$oauthtoken' --password=$NGC_API_KEY

Namespace

You can choose to deploy to whichever namespace is appropriate, but for documentation purposes we will deploy to a namespace named nrem.

kubectl create namespace nrem

Creating Secrets

Use the following script below to create the expected secrets for this helm chart.

DOCKER_CONFIG='{"auths":{"nvcr.io":{"username":"$oauthtoken", "password":"'${NGC_API_KEY}'" }}}'
echo -n $DOCKER_CONFIG | base64 -w0
NGC_REGISTRY_PASSWORD=$(echo -n $DOCKER_CONFIG | base64 -w0 )

cat <<EOF > imagepull.yaml
apiVersion: v1
kind: Secret
metadata:
  name: nvcrimagepullsecret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ${NGC_REGISTRY_PASSWORD}
EOF

cat <<EOF > ngc-cli.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ngc-api
type: Opaque
data:
  NGC_CLI_API_KEY: ${NGC_API_KEY}
EOF

kubectl apply -n nrem -f imagepull.yaml
kubectl apply -n nrem -f ngc-cli.yaml

Configuration Considerations

The following deployment commands will by default create a single deployment with one replica using the NV-EmbedQA-E5-V5 model. The following options can be used to make modifications to the behavior. See below for all parameters.

• image.repository -- The container (Embedder NIM) to deploy
• image.tag -- The version of that container (Embedder NIM)
• Storage options, based on the environment and cluster in use
• resources -- Use this option when a model requires more than the default of one GPU. See below for support matrix and resource requirements.
• env -- Which is an array of environment variables presented to the container, if advanced configuration is needed

Deploying

Basic deploymnet

helm upgrade --install \
  --namespace nrem \
  nemo-embedder \
  text-embedding-nim-1.0.0.tgz

After deploying check the pods to ensure that it is running, initial image pull and model download can take upwards of 15 minutes.

kubectl get pods -n nrem

The pod should eventually end up in the running state.

NAME              READY   STATUS    RESTARTS   AGE
nemo-embedding-ms-0   1/1     Running   0          8m44s

Storage

Storage is a particular concern when setting up NIMs. Models can be quite large, and you can fill disk downloading things to emptyDirs or other locations around your pod image. It is best to ensure you have persistent storage of some kind mounted on your pod.

This chart supports two general categories:

1. Persistent Volume Claims (enabled with persistence.enabled)
2. hostPath (enabled with persistences.hostPath)

The chart will default to use the standard storage class and will create a PersistentVolume and a PersistentVolumeClaim.

If you do not have a Storage Class Provisioner which creates PersistentVolumes automatically, then set the value persistence.createPV=true. This is also necessary when you are using persistence.hostPath on minikube.

If you have an existing PersistentVolumeClaim where you'd like the models to be stored at, then pass that value in at persistence.exsitingClaimName.

See additional options below in Parameters.

Recommended configuration for Minikube

Minkube will create a hostPath based PV and PVC by default with this chart. We recommend that you add the following to your helm commands.

--set persistence.class=standard

Deploying Snowflake Arctic Embedding

Run the helm command with the following parameters, update your version in image.tag:

helm upgrade --install \
  --namespace nrem \
  --set image.repository=nvcr.io/nim/snowflake/arctic-embed-l \
  --set image.tag=1.0.0 \
  nemo-embedder \
  text-embedding-nim-1.0.0.tgz

Deploying Mistral 7B

Create a values files for the resource requirements of the 7B Model:

# values-mistral.yaml
resources:
  limits:
    ephemeral-storage: 28Gi
    nvidia.com/gpu: 1
    memory: 32Gi
    cpu: "16000m"
  requests:
    ephemeral-storage: 28Gi
    nvidia.com/gpu: 1
    memory: 16Gi
    cpu: "4000m"

Then deploy the model:

helm upgrade --install \
  --namespace nrem \
  -f values-mistral.yaml \
  --set image.repository=nvcr.io/nim/nvidia/nv-embedqa-mistral-7b-v2 \
  --set image.tag=1.0.0 \
  nemo-embedder \
  text-embedding-nim-1.0.0.tgz

Running inference

In the previous example the API endpoint is exposed on port 8080 through the Kubernetes service of the default type with no ingress, since authentication is not handled by the NIM itself. The following commands assume the NV-EmbedQA-E5-V5 model was deployed.

Adjust the "model" value in the request JSON body to use a different model.

Use the following command to port-forward the service to your local machine to test inference.

kubectl port-forward -n nrem service/nemo-embedding-ms 8080:8080

Then try a request:

curl -X 'POST' \
  'http://localhost:8080/v1/embeddings' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "hello world",
    "model": "nv-embedqa-e5-v5",
    "input_type": "passage"
  }'

Parameters

Deployment parameters

Autoscaling parameters

Values used for autoscaling. If autoscaling is not enabled, these are ignored. They should be overridden on a per-model basis based on quality-of-service metrics as well as cost metrics. This isn't recommended except with usage of the custom metrics API using something like the prometheus-adapter. Standard metrics of CPU and memory are of limited use in scaling NIM

Ingress parameters

Probe parameters

Storage parameters

Service parameters

Monitoring

Opentelemetry configurations can be found here

Support Matrix

Hardware

NVIDIA NIMs will run on any NVIDIA GPU, as long as the GPU has sufficient memory, or on multiple, homogeneous NVIDIA GPUs with sufficient aggregate memory and CUDA compute capability > 7.0 or higher (8.0 for bfloat16). Some model/GPU combinations are optimized. See the following Supported Models section for further information.

Software

• Linux operating systems (Ubuntu 20.04 or later recommended)
• Docker >= 23.0.1
• NVIDIA Driver >= 535

Supported Models

These models are optimized using TRT-LLM and are available as pre-built, optimized engines on NGC.

You are responsible for ensuring that your use of NVIDIA AI Foundation Models complies with all applicable laws.

NV-EmbedQA-Mistral-7b-v2

Load this NIM by adding the following to the helm upgrade command.

--set image.repository=nvcr.io/nim/nvidia/nv-embedqa-mistral-7b-v2 \
--set image.tag=1.0.0 \

Optimized configurations

The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model; Profile is for what the model is optimized.

Non-optimized configuration

The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.

NV-EmbedQA-E5-V5

Load this NIM by adding the following to the helm upgrade command.

--set image.repository=nvcr.io/nim/nvidia/nv-embedqa-e5-v2 \
--set image.tag=1.0.0 \

Optimized configurations

The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model; Profile is for what the model is optimized.

Non-optimized configuration

The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.

Snowflake Arctic

Load this NIM by adding the following to the helm upgrade command.

--set image.repository=nvcr.io/nim/nvidia/arctic-embed-l \
--set image.tag=1.0.0 \

Optimized configurations

The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model; Profile is for what the model is optimized.

Non-optimized configuration

The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.