Overview

The Dynamo Frontend container is a lightweight, framework-less image designed to deploy and run CPU-bound frontend components without requiring CUDA or backend engine dependencies (vLLM, SGLang, TensorRT-LLM). It enables flexible deployment topologies by separating the frontend from inference backends, and integrates with the Gateway API Inference Extension (GAIE) for Kubernetes-native request routing.
Quick Links: Key Components | Release Info | Getting Started | Support

Key Components

• OpenAI-Compatible Frontend: HTTP API server compatible with OpenAI's chat completions and completions endpoints, handling request preprocessing, validation, and response formatting.
• Endpoint Picker (EPP): InferenceScheduler with routing, flow control, and request management for intelligent backend selection. Integrates with Gateway API Inference Extension (GAIE) for Kubernetes-native load balancing.
• Request Router: Routes requests to appropriate backend workers based on prefix matching, load, and KV cache state.
• Mock Workers: Test Dynamo components without GPU backends for development, CI/CD, and validation workflows.
• Kubernetes-Native Infrastructure: Service discovery via EndpointSlices and transport-agnostic request plane (TCP default) enable deployment without etcd or NATS dependencies.
  For more information about Dynamo frontend and GAIE, please refer to the GitHub repository and GAIE documentation.

Release Info

For the complete release history including architecture details, see the Release Artifacts page.
Pre-built containers are available for both x86_64 (AMD64) and ARM64 architectures.

Getting Started

1. Select the Tags tab and locate the container image release that you want to run.
2. In the Pull Tag column, click the icon to copy the docker pull command.
3. Open a command prompt and paste the pull command. Ensure the pull completes successfully.
4. Run the container:

docker run -it nvcr.io/nvidia/ai-dynamo/dynamo-frontend:<version>

For next steps, including deployment options and examples, please refer to the Dynamo README.

Use Cases

• Separated Frontend Deployment: Run frontend on CPU nodes while backends run on GPU nodes
• Gateway API Integration: Use with GAIE for Kubernetes-native inference routing
• Development & Testing: Test Dynamo pipelines with mock workers without GPU resources
• CI/CD Validation: Validate configurations and routing logic in automated pipelines

Support Matrix

Please refer to the support matrix for detailed hardware and architecture support.

Related Containers

• vLLM Runtime - vLLM inference backend
• SGLang Runtime - SGLang inference backend
• TensorRT-LLM Runtime - TensorRT-LLM inference backend
• Kubernetes Operator - K8s deployment automation

License

NVIDIA Dynamo is released under the Apache-2.0 open-source license, making it freely available for development, research, and deployment.

Technical Support

• Documentation: Dynamo Documentation
• GAIE Guide: Gateway API Inference Extension
• GitHub Issues: Dynamo GitHub Issues
• Release Notes: GitHub Releases