Overview

Dynamo is an open-source, high-throughput, low-latency inference framework for serving generative AI and reasoning models in multi-node distributed environments. Built in Rust for performance and Python for extensibility, Dynamo is inference-engine agnostic — supporting SGLang, TensorRT-LLM, and vLLM — and fully open source under the Apache 2.0 license.

Key Capabilities

• Disaggregated Prefill and Decode — Separate prefill and decode phases across GPUs to maximize throughput while meeting latency targets
• KV-Aware Request Routing — Route requests to workers that already hold relevant KV cache, eliminating redundant recomputation
• Dynamic GPU Scheduling — Automatically adjust GPU allocation based on fluctuating demand via an SLA-based Planner
• Accelerated Data Transfer (NIXL) — Reduce inter-node inference latency with the NVIDIA Transfer Library
• KV Cache Offloading — Leverage multiple memory hierarchies (GPU HBM, CPU, SSD) for higher effective throughput
• Multimodal Inference — Serve models that process text, images, video, and audio
• Tool Calling and Agentic Workflows — Native support for function-calling models across all backends

The Dynamo Collection

Containers

• Dynamo SGLang Runtime — Dynamo with the SGLang inference engine, optimized for high-throughput serving
• Dynamo TensorRT-LLM Runtime — Dynamo with the TensorRT-LLM inference engine, optimized for maximum single-engine performance
• Dynamo vLLM Runtime — Dynamo with the vLLM inference engine, offering the broadest feature coverage including LoRA, speculative decoding, and multimodal support
• Dynamo Frontend — A high-performance, OpenAI-compatible API gateway written in Rust with built-in prompt templating, tokenization, and request routing
• Dynamo Kubernetes Operator — Automates deployment, configuration, and lifecycle management of Dynamo inference graphs in Kubernetes
• Dynamo Snapshot Agent — Enables fast GPU worker recovery via CRIU checkpointing

Helm Charts

• Dynamo Platform — Deploys the Dynamo operator and its dependencies (including CRDs)
• Snapshot — DaemonSet for fast GPU worker recovery

Python Wheels ai-dynamo (PyPI) | ai-dynamo-runtime (PyPI) | kvbm (PyPI)

Rust Crates dynamo-runtime | dynamo-llm | all crates

Source GItHub Repositories

Getting Started

• Introduction — What Dynamo is, how it works, and when to use it
• Quick Start — Run Dynamo locally or on Kubernetes in minutes
• Support Matrix — Supported GPUs, operating systems, CUDA versions, and backend dependencies
• Feature Matrix — Backend feature compatibility across SGLang, TensorRT-LLM, and vLLM
• Release Artifacts — Container images, Python wheels, Helm charts, and Rust crates for every release

License

Dynamo is released under the Apache 2.0 license, freely available for development, research, and deployment.