What is the NeMo Framework Megatron Backend Container?

NVIDIA NeMo™ is a comprehensive software suite to build, monitor, and optimize AI agents across their lifecycle at enterprise scale.

NVIDIA NeMo framework, part of NVIDIA NeMo suite, is an end-to-end platform for pre-training, post-training, and reinforcement learning of multi-modal generative AI models across cloud, on-prem, and hybrid setups. NeMo framework is designed to enable experimentation and research with flexibility, and production-grade training with scale and speed. It utilizes state-of-the-art NVIDIA technologies to facilitate a complete workflow from distributed data processing to training of large-scale bespoke models using advanced parallelism strategies such as tensor, pipeline, data, expert, and context parallelism, to deployment on an infrastructure of your choice, be it on-premises or in the cloud.

For enterprises running their business on AI, NVIDIA AI Enterprise provides a production-grade, secure, end-to-end software platform that includes NeMo as well as generative AI reference applications and enterprise support to streamline adoption. Now organizations can integrate AI into their operations, streamlining processes, enhancing decision-making capabilities, and ultimately driving greater value.

What You Get with NVIDIA NeMo Framework Megatron Backend Container

At the heart of the NeMo framework lies the unification of distributed training and advanced parallelism. NeMo expertly uses GPU resources and memory across nodes, leading to groundbreaking efficiency gains. By dividing the model and training data, NeMo enables seamless multi-node and multi-GPU training, significantly reducing training time and enhancing overall productivity. A standout feature of NeMo is its incorporation of various parallelism and memory saving techniques:

Parallelism Techniques

• Data Parallelism
• Megatron FSDP
• Tensor Parallelism
• Pipeline Parallelism
• Sequence Parallelism
• Expert Parallelism
• Context Parallelism
• Generalized Tensor Parallelism

Memory-Saving Techniques

• Selective Activation Recompute (SAR)
• Mixed Precision (BF16, FP8, FP4, MXFP8, NVFP4)
• Communication Overlap
• CPU offloading (Activation, Weights)
• Attention: cuDNN Fused Attention and Multi-Latent Attention (MLA)，Flash Attention (FA), Grouped Query Attention (GQA), Multi-Query Attention (MQA), Sliding Window Attention (SWA)

NeMo framework container is the leading solution to support multimodality training at scale. The platform supports language and multimodal models including Qwen, DeepSeek, Gemma, Nemotron，GLM, Minimax, Muse ,and various text-based generative AI architectures including GPT, T5, BERT, MoE, RETRO and Mamba. In addition to large language models (LLM), NeMo supports several pretrained models for Computer Vision (CV), Automatic Speech Recognition (ASR), Natural Language Processing (NLP), and Text to Speech (TTS).

The NeMo framework container offers an array of techniques to refine pretrained LLMs for specialized use cases including, LoRA, Supervised fine-tuning (SFT), Reinforcement learning from human feedback (RLHF), and more. Through these diverse customization options, NeMo offers wide-ranging flexibility that is crucial in meeting varying business requirements.

Getting Started With NVIDIA NeMo

Refer to the NeMo Framework container page for step-by-step instructions on how to get started quickly with the NeMo framework.

Questions? See the current discussions and submit a question.

Report a bug? You can report a bug.

Technical Blogs

• Building and Deploying Generative AI Models with NVIDIA NeMo Framework (video)
• Getting Started with Large Language Models (blog)
• How to Create a Custom Large Language Model (blog)
• NVIDIA SteerLM: A Simple and Practical Technique to Customize LLMs During Inference (blog)
• Unlocking the Power of Enterprise-Ready LLMs with NVIDIA NeMo (blog)
• Scale Bitwise-Deterministic Pretraining (blog)
• Discussion

Documentation

More detailed documentation is available on the Nemo framework.

Known Issues

The 26.10.container has the following known issues:

• Nemotron Super text recipes on GB200: Published SFT and PEFT commands may fail because of incorrect H100 HybridEP topology settings. A fix is in PR #6690.
• LoRA+ warmup: Unequal A/B learning rates require lr_warmup_init=0. Configurations combining a nonzero warmup initial learning rate with lora_plus_ratio != 1 are unsupported and rejected.
• BAGEL: Support remains experimental. Training recipes may fail with incompatible Megatron-Bridge and Megatron-Core versions.
• All other known issues from 26.08 are resolved in the 26.10 release.
• There is a known issue with Export-Deploy in the 26.10 release: deploy_ray_inframework.py fails during initialization when tensor parallelism (TP) is greater than 1. The rank-1 ModelWorker exits due to an assertion failure in generate_other_ranks() → _assert_primary. The fix in PR #782 is approved but not yet merged and is targeted for the upcoming patch release. In the meantime, users can apply the PR, which has been validated with the WikiText tutorial.

License

Governing Terms: Your use of the NeMo Framework is governed by the NVIDIA Software License Agreement and the Product-Specific Terms for NVIDIA AI Products.

This container contains Llama Materials governed by the Meta Llama3 Community License Agreement, and is Built with Meta Llama3.

Portions of the output-discarding checkpoint implementation included in container releases 25.04 through 26.08.01 were adapted from MindSpeed. Copyright (c) 2023, Huawei Technologies Co., Ltd Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

• Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Get Help

Enterprise Support

Get access to knowledge base articles and support cases or submit a ticket.