SGLang Model-Free NIM Experimental Collection

Description

This collection provides the assets needed to deploy a model as a model-free NIM: a generic NIM container that serves a model supplied at deploy time, and validated, model-specific recipes that carry the launch settings for each supported GPU and topology.

Model-Free NIM: unlike a model-specific NIM, the model-free NIM ships no baked-in model weights. The model is selected at deploy time through the NIM_MODEL_PATH environment variable, so a single image can serve different models and revisions without a rebuild.

Recipe: because the container is shared across models, it cannot include settings that only one model needs. Recipes fill that gap. Each recipe is a YAML file, validated by NVIDIA for one model on one GPU type and topology, that pins the engine arguments and environment variables. The NIM plans and starts the engine with those settings.

To deploy, run the container with the model mounted through NIM_MODEL_PATH and a recipe passed through NIM_CONFIG_FILE. Each recipe version matches the container tag it was validated with; the compatibility matrix in this collection lists every validated pairing of container, recipe and model.

Collection Contents

Deployment Details

Deploying from this collection requires Docker with the NVIDIA Container Toolkit and access to nvcr.io (docker login nvcr.io with your NGC API key). Download the recipe matching your model, GPU SKU, and topology from the Resources section, using the recipe version that matches your container tag; the model weights are pulled from the model URI in the table above, or mount a local copy. Then start the container with docker run, mounting the model through NIM_MODEL_PATH and passing the recipe file through NIM_CONFIG_FILE.

Step 1: Download recipe and pick one that matches your GPU

ngc registry resource download-version "nim/nvidia/<recipe-resource>:<version>"
export RECIPE="$PWD/<recipe-resource>_v<version>/<recipe-file>.yaml"

Step 2: Mount the file into container and deploy

export IMAGE="nvcr.io/nim/nvidia/sglang-model-free-nim-experimental:<version>"
export NIM_MODEL_PATH="<model-uri>"
export HF_TOKEN="hf_xxx"
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
chmod -R a+rwxt "$LOCAL_NIM_CACHE"

docker run --gpus='"device=0,1,2,3"' \
  -e NIM_MODEL_PATH=$NIM_MODEL_PATH \
  -e HF_TOKEN=$HF_TOKEN \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -v "$RECIPE:/opt/nim/recipe.yaml:ro" \
  -e NIM_CONFIG_FILE="/opt/nim/recipe.yaml" \
  -p 8000:8000 \
  $IMAGE

Replace <version> and <model-uri> with the corresponding values from the same row in the Container version table above. Select a recipe that matches the chosen version and your hardware configuration.

License/Terms of Use

GOVERNING TERMS: The NIM container is governed by the NVIDIA Software License Agreement and the Product-Specific Terms for NVIDIA AI Products.

The use of any models with this container is governed by the NVIDIA Open Model Agreement in addition to any terms which govern the specific models used.

You are responsible for ensuring that your use of any provided models complies with all applicable laws.

The recipes in this collection are validated for the following third-party models. NVIDIA does not provide these models; their use is governed by their own licenses:

• DeepSeek-V4.1-Flash-1017 (deepseek-ai/DeepSeek-V4.1-Flash-1017): MIT License
• MiniMax-M3.1-Flash (MiniMaxAI/MiniMax-M3.1-Flash): MiniMax Community License
• Beam 1.0 (reflection/Beam-1.0-501B-A23B-FP8, reflection/Beam-1.0-501B-A23B-NVFP4 and the reflection/Beam-1.0-501B-A23B-DFlash2 drafter): Apache License 2.0

Deployment Geography

Global

Security Common Vulnerabilities and Exposures (CVEs)

Please review the Security Scanning tab on the container within this collection to view the latest security scan results. For certain open-source vulnerabilities listed in the scan results, NVIDIA provides a response in the form of a Vulnerability Exploitability eXchange (VEX) document. The VEX information can be reviewed and downloaded from the Security Scanning tab.

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal teams to ensure these software components meet requirements for the relevant industry and use case and address unforeseen product misuse.

Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

Get Help

Getting started with the NIM

Deploying and integrating the NIM is straightforward thanks to our industry standard APIs. Visit the NIM Container page for release documentation, deployment guides and more NIM Container LLM.

Enterprise Support

Get access to knowledge base articles and support cases or submit a ticket.