NVIDIA NeMo™ RL accelerates reinforcement learning post-training with high-performance GPU backends, offering scalable GRPO, DPO, SFT, and distillation for multimodal models from single-node experiments to enterprise-scale clusters.
What is the NeMo RL Container?
NVIDIA NeMo™ RL is an open-source post-training library under the NVIDIA NeMo Framework for scaling reinforcement learning methods for multimodal models (LLMs, VLMs, etc.). It supports small-scale experiments and large multi-GPU, multi-node deployments. NeMo RL accelerates post-training workflows using Ray for distributed coordination and Megatron Core for high-performance training backends.
NeMo RL provides a production-grade, scalable post-training platform with multiple RL algorithms, advanced parallelism, and integration with popular model frameworks to fine-tune and align large language models.
Pull the Container
- Select Get Container on this page.
- Copy the latest tag image path.
# example docker pull nvcr.io/nvidia/nemo-rl:v0.4.0
Refer to the NeMo RL releases page for release notes and other available versions.
What You Get with NVIDIA NeMo RL Container
Built on Ray with high-performance backends, NeMo RL scales efficiently across nodes and GPUs. It supports multiple training and generation backends and includes comprehensive algorithm support:
Training Backends
- DTensor - PyTorch's next-generation distributed training with improved memory efficiency (PyTorch-native TP, SP, PP, CP, and FSDP2)
- Megatron Core - NVIDIA's high-performance training framework for scaling to large models with 6D parallelism (TP/PP/CP/SP/EP/FSDP)
Generation Backends
- vLLM - High-throughput and memory-efficient inference and serving engine
- Megatron - High-performance Megatron-native inference backend eliminating weight conversion between training and inference
Learning Algorithms
- GRPO - Group Relative Policy Optimization for efficient on-policy reinforcement learning (supports async mode for concurrent trajectory generation and training with improved GPU utilization)
- DAPO - Dual-Clip Asymmetric Policy Optimization extending GRPO with asymmetric clipping for fine-grained control
- Supervised Fine-Tuning (SFT) - Fine-tune models on instruction-following datasets
- DPO - Direct Preference Optimization for preference-based training
- Reward Modeling (RM) - Train reward models for preference learning (DTensor path)
- RLHF - Reinforcement learning from human feedback using trained reward models as environments with DTensor backend support
- On-policy Distillation - Distill knowledge from larger teacher models to smaller student models
Advanced Features
- Multi-Turn RL - Multi-turn generation and training for RL with tool use, games, and interactive scenarios
- MoE Model Support - Support for DeepSeekV3 and Qwen-3 MoE models via Megatron backend
- Sequence Packing - Sequence packing in both DTensor and Megatron Core for significant training performance gains
- Worker Isolation - Process isolation between RL actors for robust distributed training
- Environment Support - Multi-environment training and dependency isolation between components
- Hugging Face Integration - Works seamlessly with models from 1B to 70B parameters (Qwen, Llama, and more)
Performance & Scale
NeMo RL delivers exceptional performance at scale:
- Scalable Multi-Node Training - Supports training across hundreds of GPUs with efficient resource utilization
- Advanced Parallelism - 6D parallelism support (TP/PP/CP/SP/EP/FSDP) for large models and long sequences
- Optimized Generation - High-throughput inference with vLLM backend for fast rollouts
- Memory Efficiency - Sequence packing and advanced parallelism techniques reduce memory overhead
- Flexible Deployment - Works on single-node setups for experimentation and scales to multi-node clusters for production
Getting Started
Refer to the NVIDIA NeMo RL documentation for step-by-step instructions. You can also explore the source code and examples on the NVIDIA-NeMo/RL GitHub repository.
Documentation
More detailed documentation is available in the NeMo RL User Guide, including comprehensive guides for:
- Overview - Architecture and design philosophy
- Quick Start - Get up and running quickly
- Algorithms - GRPO, DAPO, SFT, DPO, RM, and On-policy Distillation guides
- Training Backends - DTensor and Megatron Core configuration
- Generation Backends - vLLM backend setup and configuration
- Cluster Setup - Deploy on Slurm or Kubernetes clusters
- Docker - Build and use Docker containers
Developer Container
For developers who want access to features that have been implemented but are not yet included in a major release, you can build your own container from the latest source code. Refer to the Docker build documentation for detailed instructions.
Support & Community
The NeMo RL framework is actively developed with regular updates. Stay informed about new releases and features by following announcements.
Need help?
- Report issues or bugs on GitHub Issues (specify version and container details)
- Ask questions in GitHub Discussions
License
GOVERNING TERMS: The software and materials are governed by the NVIDIA Software License Agreement and the Product-Specific Terms for NVIDIA AI Products.