## Model Overview

### Description:

Nemotron-3-8B-Chat-SteerLM is an 8 billion parameter
generative language model instruct-tuned on an 8B base model.
It takes input with context length up to 4,096 tokens.
The model has been customized
using the [SteerLM method](https://arxiv.org/abs/2310.05344)
developed by NVIDIA to allow for user control of model outputs during inference.

Key capabilities enabled by SteerLM:

- Dynamic steering of responses by specifying desired attributes like quality, helpfulness, and toxicity at inference
 time.
- Simplified training compared to RLHF techniques like fine-tuning and bootstrapping.

Nemotron-3-8B-SteerLM is part of Nemotron-3, a family of enterprise ready decoder-only generative text models compatible
with [NeMo Framework](https://www.nvidia.com/en-us/ai-data-science/generative-ai/nemo-framework).
For other models in this collection,
see [blogpost](https://developer.nvidia.com/blog/nvidia-ai-foundation-models-build-custom-enterprise-chatbots-and-co-pilots-with-production-ready-llms/)
NVIDIA NeMo is an end-to-end, cloud-native framework to build, customize, and deploy generative AI models anywhere.
It includes training and inferencing frameworks, guardrailing toolkits, data curation tools, and pretrained models,
offering enterprises an easy, cost-effective, and fast way to adopt generative AI.

### License

The use of this model is governed by
the [NVIDIA AI Foundation Models Community License Agreement](https://developer.nvidia.com/downloads/nv-ai-foundation-models-license)

### References(s):

* [Technical Blog](https://developer.nvidia.com/blog/nvidia-ai-foundation-models-build-custom-enterprise-chatbots-and-co-pilots-with-production-ready-llms)

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** GPT-3 

The SteerLM method involves the following key steps:

- Train an attribute prediction model on human annotated data to evaluate response quality.
- Use this model to annotate diverse datasets and enrich training data.
- Perform conditioned fine-tuning to align responses with specified combinations of attributes.
- (Optionally) Bootstrap training through model sampling and further fine-tuning.
 SteerLM-8B applies this technique on top of the open-source NVIDIA GPT model architecture. It was pretrained on
 internet-scale data and then customized
 using [OASST](https://huggingface.co/datasets/OpenAssistant/oasst1), [HH-RLHF](https://huggingface.co/datasets/Anthropic/hh-rlhf),
 [Light](https://github.com/facebookresearch/ParlAI/blob/9974b947fb2e801dc5608f495828532c2a714742/parlai/tasks/light_dialog/build.py#L14),
 a subset of permissive licensed [OpenPlatypus](https://huggingface.co/datasets/garage-bAInd/Open-Platypus), and
 some internally collected SFT data.

### Prompt Format:

**Single Turn**

```
<extra_id_0>System
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
 
<extra_id_1>User
{prompt 1}
<extra_id_1>Assistant
<extra_id_2>quality:4,understanding:4,correctness:4,coherence:4,complexity:4,verbosity:4,toxicity:0,humor:0,creativity:0,violence:0,helpfulness:4,not_appropriate:0,hate_speech:0,sexual_content:0,fails_task:0,political_content:0,moral_judgement:0,lang:en
```

**Multi-Turn or Few-shot**

```
<extra_id_0>System
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
 
<extra_id_1>User
{prompt 1}
<extra_id_1>Assistant
<extra_id_2>quality:4,understanding:4,correctness:4,coherence:4,complexity:4,verbosity:4,toxicity:0,humor:0,creativity:0,violence:0,helpfulness:4,not_appropriate:0,hate_speech:0,sexual_content:0,fails_task:0,political_content:0,moral_judgement:0,lang:en
{response 1}
<extra_id_1>User
{prompt 2}
<extra_id_1>Assistant
<extra_id_2>quality:4,understanding:4,correctness:4,coherence:4,complexity:4,verbosity:4,toxicity:0,humor:0,creativity:0,violence:0,helpfulness:4,not_appropriate:0,hate_speech:0,sexual_content:0,fails_task:0,political_content:0,moral_judgement:0,lang:en
```

Example prompt formation code:

```
PROMPT_TEMPLATE = """<extra_id_0>System
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
 
<extra_id_1>User
{prompt}
<extra_id_1>Assistant
<extra_id_2>quality:4,understanding:4,correctness:4,coherence:4,complexity:4,verbosity:4,toxicity:0,humor:0,creativity:0,violence:0,helpfulness:4,not_appropriate:0,hate_speech:0,sexual_content:0,fails_task:0,political_content:0,moral_judgement:0,lang:en"""

question = "Write a poem on NVIDIA in the style of Shakespeare"
prompt = PROMPT_TEMPLATE.format(prompt=question)
print(prompt)
```

Each of the properties (e.g. humor, toxicity…) can receive integer values in the range [0..4].

### Software Integration:

**Runtime(s):** NVIDIA AI Enterprise 
**Toolkit:** NeMo Framework 
See [this document](https://registry.ngc.nvidia.com/orgs/ea-bignlp/teams/beta-inf-prerelease/containers/infer) for
details on how to setup an inference server with the pyTriton and TensorRT-LLM backend.

**Supported Hardware Platform(s):** Hopper (H100), Ampere (A100 80GB, A100 40GB) 

### Model Version(s)

Nemotron-3-8B-chat-4k-steerlm-BF16-1

## Training & Finetuning:

### Dataset:

NVIDIA models are trained on a diverse set of public and proprietary datasets. This model was trained on a dataset
containing 3.5 Trillion tokens of text. The dataset contains 53 different human languages and 37 programming languages.
NVIDIA is committed to the responsible development of large language models and conducts reviews of all datasets
included in training.

## Evaluation

MT-Bench

| Category | Score |
|------------|-------|
| Total | 5.6 |
| Writing | 6.35 |
| Roleplay | 6.9 |
| Extraction | 5.25 |
| Stem | 7.5 |
| Humanities | 9.02 |
| Reasoning | 4.9 |
| Math | 2.0 |
| Coding | 2.9 |

## Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other 

## Intended use

This model is for users who want to customize a model’s response during inference.

### Ethical use

Technology can have a profound impact on people and the world, and NVIDIA is committed to enabling trust and
transparency in AI development. NVIDIA encourages users to adopt principles of AI ethics and trustworthiness to guide
your business decisions by following the guidelines in the NVIDIA AI Foundation Models Community License Agreement.

## Limitations

The model was trained on the data that contains toxic language and societal biases originally crawled from the Internet.
Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts.
The Model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text
producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly
offensive.

nemo-8b-steerlm

Nemotron-3-8B-Chat-SteerLM is an 8 billion parameter generative language model based on the Nemotron-3-8B base model. It has been customized for user control of model outputs during inference using the SteerLM method developed by NVIDIA.