The Dynamo vLLM runtime image is a containerized build of Dynamo + vLLM which serves as the base runtime environment for vLLM based inference with Dynamo's distributed inference framework.
Overview
The Dynamo vLLM runtime container is a pre-built, Docker-based environment designed to run NVIDIA Dynamo with the vLLM backend for high-performance, distributed large language model (LLM) inference. It packages all necessary dependencies, runtime components, and optimizations to streamline deployment and ensure consistency across development and production environments. Quick Links: Key Components | Release Info | Getting Started | Support
Key Components
- vLLM Backend: High-performance LLM inference engine with optimized attention mechanisms, continuous batching, and efficient KV cache management.
- Disaggregated Serving (P/D): Separates prefill and decode phases across specialized workers for improved throughput and latency optimization.
- Planner: SLA-aware request scheduling that routes requests based on latency targets and system load.
- KV Router: Intelligent request routing with prefix-aware caching to maximize KV cache reuse across workers.
- NIXL (KV Transfer Library): High-performance GPU-to-GPU memory transfer for distributed KV cache operations.
- OpenAI-Compatible Frontend: HTTP API server compatible with OpenAI's chat completions and completions endpoints.
- Kubernetes-Native Infrastructure: Service discovery via EndpointSlices and transport-agnostic request plane (TCP default) enable deployment without external dependencies. etcd and NATS remain available as optional alternatives for non-Kubernetes environments. For more information about Dynamo features, please refer to the GitHub repository and documentation.
Release Info
For the complete release history including vLLM versions, CUDA support, and architecture details, see the Release Artifacts page. Pre-built containers are available for both x86_64 (AMD64) and ARM64 architectures. CUDA 13 experimental variants are also available.
Getting Started
- Select the Tags tab and locate the container image release that you want to run.
- In the Pull Tag column, click the icon to copy the docker pull command.
- Open a command prompt and paste the pull command. Ensure the pull completes successfully.
- Run the container:
docker run --gpus all -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:<version>
For next steps, including deployment options and examples, please refer to the Dynamo README.
Support Matrix
Please refer to the support matrix for detailed hardware, architecture, and model support information.
Related Containers
- SGLang Runtime - High-throughput optimized backend
- TensorRT-LLM Runtime - Maximum inference performance
- Dynamo Frontend - Standalone frontend with EndpointPicker (EPP)
- Kubernetes Operator - K8s deployment automation
License
NVIDIA Dynamo is released under the Apache-2.0 open-source license, making it freely available for development, research, and deployment.
Technical Support
- Documentation: Dynamo Documentation
- GitHub Issues: Dynamo GitHub Issues
- Release Notes: GitHub Releases