vLLM Model-Free NIM (Experimental)

This collection contains the vLLM Model-Free NIM (Experimental) container and serving recipe resources for it. The container serves a model you provide at runtime, from a local directory or a Hugging Face repository, through an OpenAI-compatible API.

Artifacts

• Container: nvcr.io/nim/nvidia/vllm-model-free-nim-experimental
• Recipe resources: NIM tuning config files for validated model and GPU combinations, one file per GPU and parallelism combination. Each file's header comment names the model checkpoint, GPU, GPU memory range, and GPU count it was validated for.

See the Artifacts tab for the resources in this collection.

How NIM applies a recipe

NIM applies a recipe in one of two ways.

Automatic selection. The container includes its validated recipes. On the default nim-serve path, NIM selects a recipe when the model architecture, checkpoint parameter count, GPU model, memory per GPU, and number of visible GPUs all match a validated configuration. No extra setting is needed. debug mode skips automatic selection.

Explicit tuning config file. Set NIM_CONFIG_FILE to a recipe file to load it as the NIM tuning config. Top-level keys are vLLM settings, and the ENV section sets environment variables. A tuning config file takes priority over automatic selection.

Load a recipe file

Download a recipe resource and pick the file that matches your GPUs:

ngc registry resource download-version "nim/nvidia/<recipe-resource>:<version>"
export RECIPE="$PWD/<recipe-resource>_v<version>/<recipe-file>.yaml"

Mount the file into the container and point NIM_CONFIG_FILE at it:

export IMAGE=nvcr.io/nim/nvidia/vllm-model-free-nim-experimental:<version>
export MODEL_DIR=/path/to/model

docker run --rm \
  --gpus <gpu-count> \
  --ipc=host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -p 8000:8000 \
  -v nim-cache:/opt/nim/.cache \
  -v "$MODEL_DIR:/models/model:ro" \
  -v "$RECIPE:/opt/nim/recipe.yaml:ro" \
  -e NIM_MODEL_PATH=/models/model \
  -e NIM_CONFIG_FILE=/opt/nim/recipe.yaml \
  "$IMAGE" \
  nim-serve

Use the GPU count stated in the recipe file. MODEL_DIR must be readable by UID 1000 inside the container.

To load the model from Hugging Face instead, remove the model mount, export HF_TOKEN on the host, and replace the NIM_MODEL_PATH line with:

-e NIM_MODEL_PATH=hf://<organization>/<repository>
-e HF_TOKEN

Override a setting

Settings are resolved in this order, from lowest to highest priority: automatic recipe, tuning config file, environment variables, NIM_PASSTHROUGH_ARGS, and nim-serve command-line options. To change one value, pass it on the command line, for example nim-serve --max-num-seqs 128. Changing recipe settings moves the deployment outside the validated configuration.

Verify the service

curl --fail http://localhost:8000/v1/health/ready
curl --fail http://localhost:8000/v1/models

The id field in the /v1/models response is the model name to use in API requests.