Kimi K3.1 Pro Serving Recipes

NIM tuning config files for Kimi K3.1 Pro on nvcr.io/nim/nvidia/vllm-model-free-nim-experimental:2.1.6-variant-kimi-k3.1-pro. Each file covers one GPU, node count, and node role. All files use a 1,048,576-token context, no speculative decoding, and tool_strict_level: parameter.

The header comment in each file names the image, GPU, parallelism, and checkpoint it was prepared for.

Download the recipes

ngc registry resource download-version "nim/nvidia/kimi-k3.1-pro-recipe:2.1.6-variant-kimi-k3.1-pro"
export RECIPE_DIR="$PWD/kimi-k3.1-pro-recipe_v2.1.6-variant-kimi-k3.1-pro"
chmod 0644 "$RECIPE_DIR"/*.yaml

NGC CLI saves downloaded files with mode 0600. The container runs as UID 1000, so make the recipe files and the model directory readable by that user, and the cache directory writable by it.

Get the checkpoint

The released checkpoint is moonshotai/Kimi-K3.1-Pro on Hugging Face (MXFP4, about 1.6 TB). The B300 recipe uses this checkpoint. The H20-3e recipes were validated with a pre-release checkpoint.

The recipes run with HF_HUB_OFFLINE=1, so download the checkpoint to a local directory on every node first and mount it as MODEL_DIR:

export MODEL_DIR=/path/to/Kimi-K3.1-Pro
huggingface-cli download moonshotai/Kimi-K3.1-Pro --local-dir "$MODEL_DIR"

Start a multi-node H20-3e deployment

Use the head file on the primary node and the matching worker file on each other node. For the 4-node deployment, use the worker1, worker2, and worker3 files in node-rank order. Run on every node:

export IMAGE=nvcr.io/nim/nvidia/vllm-model-free-nim-experimental:2.1.6-variant-kimi-k3.1-pro

docker run --rm <GPU-and-fabric-device-options> \
  --network host --ipc private --shm-size 32g --ulimit memlock=-1:-1 \
  -v "$MODEL_DIR:/model:ro" -v "$RECIPE_DIR:/recipes:ro" \
  -v "$CACHE_DIR:/nim-cache" \
  -e BACKEND_TYPE=vllm -e NIM_MODEL_PATH=/model -e NIM_WORKSPACE_PATH=/model \
  -e NIM_CONFIG_FILE=/recipes/<selected-role-file>.yaml \
  -e NIM_CACHE_PATH=/nim-cache -e NIM_RUNTIME_CACHE_PATH=/nim-cache/runtime-cache \
  -e NIM_TRUST_CUSTOM_CODE=1 \
  -e NIM_PRIMARY_NODE=<primary-node-address> -e NIM_NODE_MANAGER_PORT=29541 \
  -e NIM_BACKEND_HOST=127.0.0.1 -e NIM_BACKEND_PORT=38082 \
  -e NIM_SERVER_PORT=38083 "$IMAGE" nim-serve

Select eight GPUs per node with --gpus '"device=GPU-UUID1,...,GPU-UUID8"' and map /dev/infiniband/rdma_cm plus that host's uverbs devices with --device. Use the host's RDMA fabric setup; NIM derives the communication settings. All nodes use the primary node's address.

Start a single-node B300 deployment

Use the moonshotai/Kimi-K3.1-Pro checkpoint. The header comment of kimi-k3.1-pro-mxfp4-b300-8gpu-nn1-tp8-pp1.yaml contains the complete docker run command; set MODEL_DIR to the checkpoint directory and run it from the recipe directory. Keep the -e settings in that command: they apply during initialization, before NIM loads the YAML.

Override a setting

Settings are resolved in this order, from lowest to highest priority: tuning config file, environment variables, NIM_PASSTHROUGH_ARGS, and nim-serve command-line options. Changing recipe settings moves the deployment outside the validated configuration.

Verify the service

curl --fail http://<primary-node-address>:<public-port>/v1/health/ready
curl --fail http://<primary-node-address>:<public-port>/v1/models

The recipes do not set a served-model alias. Use the id returned by /v1/models as the model field in API requests.

Get Help

NVIDIA Developer Community Forum

Get access to community knowledge base articles and support cases (https://forums.developer.nvidia.com/)