## Model Overview

### Description:
NV-Llama2-70B-RLHF-Chat generates text and code in response to prompts. It is a 70 billion parameter generative language model instruct-tuned on [LLama2-70B model](https://huggingface.co/meta-llama/Llama-2-70b). It takes input with context length up to 4,096 tokens. The model has been further fine-tuned for instruction following using Supervised Fine-tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF),  achieving 7.59 on [MT-Bench](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard) and demonstrating strong performance on academic benchmarks.

NV-Llama2-70B-RLHF-Chat is trained with [NVIDIA NeMo-Aligner](https://github.com/NVIDIA/NeMo-Aligner), a scalable toolkit for efficient model alignment. The toolkit has support for state of the art model alignment algorithms such as Reinforcement Learning from Human Feedback (RLHF) and SteerLM. RLHF enables users to align language models to be more safe, harmless and helpful. Enterprises can do end-to-end RLHF on a wide range of model sizes. They can also take advantage of all the parallelism to ensure their model alignment is done in a performant and resource-efficient manner.


### License:
​
The use of this model is governed by the [NVIDIA AI Foundation Models Community License Agreement](https://developer.nvidia.com/downloads/nv-ai-foundation-models-license). Llama 2 is licensed under the [LLAMA 2 Community License](https://ai.meta.com/llama/license/), Copyright © Meta Platforms, Inc. All Rights Reserved.


### Model Architecture:
​
**Architecture Type:** Transformer

​
**Network Architecture:** Llama2
​
### Input:

**Input Format:** Text

​
**Input Parameters:** Temperature, Top P = 1, Max Input Tokens: 3072

### Output:

**Output Format:** Text

​
**Output Parameters:** Max Output Tokens: 1024


### Software Integration:

**Supported Hardware Platform(s):** Ampere/Turing, Hopper

**Test Hardware:** H100, A100 80GB, A100 40GB

**Supported Operating System(s):** Linux
### Model Version:
V1.0


## Training: 
### Datasets:
We first conduct Supervised Fine-tuning on the Llama2-70B model with a diverse set of public and proprietary datasets, all of which are available for commercial use. Initializing with the SFT-ed model, we further instruct-tune it using RLHF with Anthropic’s HH-RLHFdataset.

**Public Dataset Links:**
[Open Assistant](https://huggingface.co/datasets/OpenAssistant/oasst1),
[HH-RLHF](https://huggingface.co/datasets/Anthropic/hh-rlhf)

**Properties (Quantity, Dataset Descriptions, Sensor(s)):**
Open Assistant is a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292 quality ratings, resulting in over 10,000 fully annotated conversation trees.


HH-RLHF is a human preference data about helpfulness and harmlessness from Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. It contains 161K prompts with chosen and rejected responses.
The internal, private dataset contains about 25k samples of prompts covering generation, classification, brainstorming, extraction, closed QA (question & answer), open QA, rewrite and summarization labeled by human annotators. 
## Inference:
**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) <br>
**Test Hardware:** H100, A100 


### Ethical Considerations: 

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability).

nv-llama2-70b-rlhf

NV-Llama2-70B-RLHF-Chat is a 70 billion parameter generative language model instruct-tuned on LLama2-70B model. It takes input with context length up to 4,096 tokens.