What is the NeMo RL Container?

NVIDIA NeMo™ RL is an open-source post-training library under the NVIDIA NeMo Framework for scaling reinforcement learning methods for multimodal models (LLMs, VLMs, etc.). It supports small-scale experiments and large multi-GPU, multi-node deployments. NeMo RL accelerates post-training workflows using Ray for distributed coordination and Megatron Core for high-performance training backends.

NeMo RL provides a production-grade, scalable post-training platform with multiple RL algorithms, advanced parallelism, and integration with popular model frameworks to fine-tune and align large language models.

Pull the Container

1. Select Get Container on this page.
2. Copy the latest tag image path.
     # example
     docker pull nvcr.io/nvidia/nemo-rl:v0.4.0
   

Refer to the NeMo RL releases page for release notes and other available versions.

What You Get with NVIDIA NeMo RL Container

Built on Ray with high-performance backends, NeMo RL scales efficiently across nodes and GPUs. It supports multiple training and generation backends and includes comprehensive algorithm support:

Training Backends

• DTensor - PyTorch's next-generation distributed training with improved memory efficiency (PyTorch-native TP, SP, PP, CP, and FSDP2)
• Megatron Core - NVIDIA's high-performance training framework for scaling to large models with 6D parallelism (TP/PP/CP/SP/EP/FSDP)

Generation Backends

• vLLM - High-throughput and memory-efficient inference and serving engine
• Megatron - High-performance Megatron-native inference backend eliminating weight conversion between training and inference

Learning Algorithms

• GRPO - Group Relative Policy Optimization for efficient on-policy reinforcement learning (supports async mode for concurrent trajectory generation and training with improved GPU utilization)
• DAPO - Dual-Clip Asymmetric Policy Optimization extending GRPO with asymmetric clipping for fine-grained control
• Supervised Fine-Tuning (SFT) - Fine-tune models on instruction-following datasets
• DPO - Direct Preference Optimization for preference-based training
• Reward Modeling (RM) - Train reward models for preference learning (DTensor path)
• RLHF - Reinforcement learning from human feedback using trained reward models as environments with DTensor backend support
• On-policy Distillation - Distill knowledge from larger teacher models to smaller student models

Advanced Features

• Multi-Turn RL - Multi-turn generation and training for RL with tool use, games, and interactive scenarios
• MoE Model Support - Support for DeepSeekV3 and Qwen-3 MoE models via Megatron backend
• Sequence Packing - Sequence packing in both DTensor and Megatron Core for significant training performance gains
• Worker Isolation - Process isolation between RL actors for robust distributed training
• Environment Support - Multi-environment training and dependency isolation between components
• Hugging Face Integration - Works seamlessly with models from 1B to 70B parameters (Qwen, Llama, and more)

Performance & Scale

NeMo RL delivers exceptional performance at scale:

• Scalable Multi-Node Training - Supports training across hundreds of GPUs with efficient resource utilization
• Advanced Parallelism - 6D parallelism support (TP/PP/CP/SP/EP/FSDP) for large models and long sequences
• Optimized Generation - High-throughput inference with vLLM backend for fast rollouts
• Memory Efficiency - Sequence packing and advanced parallelism techniques reduce memory overhead
• Flexible Deployment - Works on single-node setups for experimentation and scales to multi-node clusters for production

Getting Started

Refer to the NVIDIA NeMo RL documentation for step-by-step instructions. You can also explore the source code and examples on the NVIDIA-NeMo/RL GitHub repository.

Documentation

More detailed documentation is available in the NeMo RL User Guide, including comprehensive guides for:

• Overview - Architecture and design philosophy
• Quick Start - Get up and running quickly
• Algorithms - GRPO, DAPO, SFT, DPO, RM, and On-policy Distillation guides
• Training Backends - DTensor and Megatron Core configuration
• Generation Backends - vLLM backend setup and configuration
• Cluster Setup - Deploy on Slurm or Kubernetes clusters
• Docker - Build and use Docker containers

Developer Container

For developers who want access to features that have been implemented but are not yet included in a major release, you can build your own container from the latest source code. Refer to the Docker build documentation for detailed instructions.

Support & Community

The NeMo RL framework is actively developed with regular updates. Stay informed about new releases and features by following announcements.

Need help?

• Report issues or bugs on GitHub Issues (specify version and container details)
• Ask questions in GitHub Discussions

License

GOVERNING TERMS: The software and materials are governed by the NVIDIA Software License Agreement and the Product-Specific Terms for NVIDIA AI Products.