## Model Overview

### Description:

Nemotron-3-8B-Chat-SteerLM is an 8 billion parameter generative language model based on the Nemotron-3-8B base model. It has been customized using the [SteerLM method](https://arxiv.org/abs/2310.05344) developed by NVIDIA to allow for user control of model outputs during inference.
Key capabilities enabled by SteerLM:
- Dynamic steering of responses by specifying desired attributes like quality, helpfulness, and toxicity `at inference time`.
- Simplified training compared to RLHF techniques like fine-tuning and bootstrapping.

NV-GPT-8B-SteerLM is part of NV-GPT, a family of enterprise ready decoder-only generative text models compatible with [NeMo Framework](https://www.nvidia.com/en-us/ai-data-science/generative-ai/nemo-framework).
NVIDIA NeMo is an end-to-end, cloud-native framework to build, customize, and deploy generative AI models anywhere. It includes training and inferencing frameworks, guardrailing toolkits, data curation tools, and pretrained models, offering enterprises an easy, cost-effective, and fast way to adopt generative AI.

### Terms of use

The use of this model is governed by the NVIDIA NeMo Foundational Models Community License Agreement

### References(s):

* Announcement Blog coming soon

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** GPT-3 
**Model version:** N/A 

### Input:

**Input Format:** Text 
**Input Parameters:** Temperature, TopP, TopK 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** Text 
**Output Parameters:** Max output tokens 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** NVIDIA AI Enterprise 
**Toolkit:** NeMo Framework 
**Supported Hardware Platform(s):** Hopper, Ampere 
**Supported Operating System(s):** Linux 

## Training & Finetuning:

### Dataset:

NVIDIA models are trained on a diverse set of public and proprietary datasets. This model was trained on a dataset containing 3.5 Trillion tokens of text. The dataset contains 53 different human languages and 37 programming languages. NVIDIA is committed to the responsible development of large language models and conducts reviews of all datasets included in training.

### Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

nemo-8b-steerlm

Nemotron-3-8B-Chat-SteerLM is an 8 billion parameter generative language model based on the Nemotron-3-8B base model. It has been customized for user control of model outputs during inference using the SteerLM method developed by NVIDIA.

Text Generation

## Model Overview

### Description:

Llama 2 SteerLM Chat is a large language model customized using the [SteerLM method](https://arxiv.org/abs/2310.05344)
developed by NVIDIA to allow for user control of model outputs during inference.

### Terms of use

By accessing this model, you are agreeing to the LLama 2 terms and conditions of the [license](https://l.facebook.com/l.php?u=https%3A%2F%2Fai.meta.com%2Fllama%2Flicense&h=AT0DpDKXSE-Yi_pulcu4-5oKNHyiayGrLbQqW7qI_4Ez5y-67Nmyyu38D5NcEmGSifLOtDZAJaQOPMGJg8zCFDJjbYcyJZQPawxWU3-GaVdgStbboYT00yqZvyrxP1DVlbfAog), [acceptable use policy](https://l.facebook.com/l.php?u=https%3A%2F%2Fai.meta.com%2Fllama%2Fuse-policy&h=AT17YXIWa1dxwuzDq96w614_oshaUCWIQYHHHf9BgC0X5UK39DW2tpSP6NzkeadQVs0XBaNlFSo4nydTUcKB5r-Kn4OLs_5IbMEZYAf5hUvyzIwx8-mLom2ic1vw5rSPFUoc9A) and [Meta’s privacy policy](https://www.facebook.com/privacy/policy/)

### References(s):

* [SteerLM method](https://arxiv.org/abs/2310.05344)
* [Llama 2: Open Foundation and Fine-Tuned Chat Models Paper](https://scontent-lhr8-2.xx.fbcdn.net/v/t39.2365-6/10000000_662098952474184_2584067087619170692_n.pdf?_nc_cat=105&ccb=1-7&_nc_sid=3c67a6&_nc_ohc=SvLw337Y0sQAX8zTcqT&_nc_ht=scontent-lhr8-2.xx&oh=00_AfC2GBa-ZPdVLSxOnnjuTZHTDu51NZUcR6KubtLe5tEUbw&oe=64D0333F) 
* [Meta's Llama 2 Webpage](https://ai.meta.com/llama/) 
* [Meta's Llama 2 Model Card](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md) 

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** Llama 2 

### Input:

**Input Format:** Text 
**Input Parameters:** Temperature, Top P = 1, Max Input Tokens: 3072 

### Output:

**Output Format:** Text 
**Output Parameters:** Max Output Tokens: 1024 

### Software Integration:

**Supported Hardware Platform(s):** Ampere/Turing, Hopper 
**Test Hardware** H100, A100 80GB, A100 40GB 
**Supported Operating System(s):** Linux 

### Model Version:

v1.0

## Training & Evaluation:

### Datasets:

**Link:**
* [Open Assistant](https://huggingface.co/datasets/OpenAssistant/oasst1)
* [HelpSteer](https://huggingface.co/datasets/nvidia/HelpSteer)

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
* Open Assistant is a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292 quality ratings, resulting in over 10,000 fully annotated conversation trees.
* HelpSteer is a hybrid human and synthetically-generated, human-annotated dataset of 37,120 prompt-responses built to make more models helpful, factually-correct, coherent, and customizable based on complexity and verbosity.

## Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** H100, A100 80GB, A100 40GB 

### Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

llama2-70b-steerlm

Llama 2 SteerLM Chat is a large language model, aligned using the SteerLM technique developed by NVIDIA. This allows you to adjust the preferred style of response to attributes (such as creativity, complexity and verbosity) at inference time.

## Model Overview

### Description:

Llama 2 is a large language AI model comprising a collection of models capable of generating text and code in response to prompts.

### Terms of use

By accessing this model, you are agreeing to the Llama 2 terms and conditions of the [license](https://l.facebook.com/l.php?u=https%3A%2F%2Fai.meta.com%2Fllama%2Flicense&h=AT0DpDKXSE-Yi_pulcu4-5oKNHyiayGrLbQqW7qI_4Ez5y-67Nmyyu38D5NcEmGSifLOtDZAJaQOPMGJg8zCFDJjbYcyJZQPawxWU3-GaVdgStbboYT00yqZvyrxP1DVlbfAog), [acceptable use policy](https://l.facebook.com/l.php?u=https%3A%2F%2Fai.meta.com%2Fllama%2Fuse-policy&h=AT17YXIWa1dxwuzDq96w614_oshaUCWIQYHHHf9BgC0X5UK39DW2tpSP6NzkeadQVs0XBaNlFSo4nydTUcKB5r-Kn4OLs_5IbMEZYAf5hUvyzIwx8-mLom2ic1vw5rSPFUoc9A) and [Meta’s privacy policy](https://www.facebook.com/privacy/policy/)

### References(s):

* [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://scontent-lhr8-2.xx.fbcdn.net/v/t39.2365-6/10000000_662098952474184_2584067087619170692_n.pdf?_nc_cat=105&ccb=1-7&_nc_sid=3c67a6&_nc_ohc=SvLw337Y0sQAX8zTcqT&_nc_ht=scontent-lhr8-2.xx&oh=00_AfC2GBa-ZPdVLSxOnnjuTZHTDu51NZUcR6KubtLe5tEUbw&oe=64D0333F) paper 
* [Meta's Llama 2](https://ai.meta.com/llama/) webpage 
* [Meta's Llama 2 Model Card](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md) webpage 

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** Llama 2 
**Model version:** N/A 

### Input:

**Input Format:** Text 
**Input Parameters:** Temperature, TopP 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** Text 
**Output Parameters:** Max output tokens 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** N/A 
**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

## Training & Finetuning:

### Dataset:

**Link:**
* [Falcon RefinedWeb](https://arxiv.org/abs/2306.01116)
* [BigCode - The Stack (StarCoder)](https://huggingface.co/datasets/bigcode/the-stack)
* [Wikipedia](https://www.tensorflow.org/datasets/catalog/wikipedia)
* [ArXiv](https://arxiv.org/pdf/2101.00027.pdf)
* [Books3](https://arxiv.org/pdf/2101.00027.pdf)
* [StackExchange](https://archive.org/details/stackexchange) 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
* Falcon RefinedWeb is a massive English web dataset containing 500-650GT depending on the used tokenizer
* The Stack dataset is a collection of source code in over 300 programming languages
* Wikipedia is 20.54GB of online encyclopedia written and maintained by a community of volunteers
* The Pile: ArXiv is 56.21GB of ArXiv papers contents
* The Pile: Books3 is 100.96GB dataset of books derived from a copy of the contents of the Bibliotik private tracker made available by Shawn Presser
* StackExchange is an anonymized dump of all user-contributed content on the Stack Exchange network where each site is formatted as a separate archive consisting of zipped XML files

**Dataset License:**
Free for research and commercial use.

## Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

llama2-70b

Llama 2 is a large language AI model capable of generating text and code in response to prompts.

## Model Overview

### Description:

SDXL is a latent diffusion model for text-to-image synthesis. Compared to previous versions of Stable Diffusion, SDXL leverages a three times larger UNet backbone: The increase of model parameters is mainly due to more attention blocks and a larger cross-attention context as SDXL uses a second text encoder.

### Terms of use

By accessing this model, you are agreeing to the SDXL 1.0 terms and conditions of the [license](https://github.com/Stability-AI/generative-models/blob/main/model_licenses/LICENSE-SDXL1.0), [acceptable use policy](https://github.com/Stability-AI/stablediffusion/blob/main/LICENSE) and [stability.ai privacy policy](https://platform.stability.ai/legal/privacy-policy)

### References(s):

* [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://arxiv.org/abs/2307.01952) paper 
* [Stability-AI](https://github.com/Stability-AI/generative-models) repo 
* [Stability-AI's SDXL Model Card](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) webpage 

### Model Architecture:

**Architecture Type:** Transformer and Convolutional Neural Network (CNN) 
**Network Architecture:** UNet + attention blocks 
**Model version:** SDXL 1.0 

### Input:

**Input Format:** Text 
**Input Parameters:** scheduler type, denoising steps, classifier-free guidance 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** Red, Green, Blue (RGB) Image 
**Output Parameters:** 2D 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** N/A 
**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

## Training & Finetuning:

### Dataset:

**Link:** [LAION 5B](https://laion.ai/blog/laion-5b/) 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
The dataset consists of 5.85 billion CLIP-filtered image-text pairs. 

**Dataset License:**
* [CreativeML Open RAIL-M license](https://huggingface.co/spaces/CompVis/stable-diffusion-license) 

## Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

sdxl

Stable Diffusion XL (SDXL) enables you to generate expressive images with shorter prompts and insert words inside images.

Image Generation

## Model Overview

### Description:

Fuyu-8B is a multi-modal transformer introduced by Adept AI. It can perform a wide range of tasks,
including image understanding, text generation, and code generation.
Architecturally, Fuyu is a vanilla decoder-only transformer - there is no image encoder.
Image patches are instead linearly projected into the first layer of the transformer, bypassing the embedding lookup.
The transformer decoder is simply treated like an image transformer (albeit with no pooling and causal attention).

### Terms of use

By accessing this model, you are agreeing to the Fuyu-8b terms and conditions of the [CC BY-NC license](https://creativecommons.org/licenses/by-nc/4.0/deed.en).

### References(s):

* [Fuyu-8B Blog Post](https://www.adept.ai/blog/fuyu-8b) by adept.ai 
* [Fuyu-8B Model Card](https://huggingface.co/adept/fuyu-8b) on Hugging Face 

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** Fuyu-8b 
**Model version:** N/A 

### Input:

**Input Format:** Red, Green, Blue (RGB) Image + Text 
**Input Parameters:** None 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** Text 
**Output Parameters:** None 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** N/A 
**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

## Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

fuyu-8b

Fuyu-8B is a multi-modal transformer that can perform a wide range of tasks, including image understanding, text generation, and code generation.

Image Conversation

## Model Overview

### Description:

Mistral-7B-Instruct is a language model that can follow instructions, complete requests, and generate creative text formats.
It is an instruct version of the Mistral-7B-v0.1 generative text model fine-tuned using a variety of publicly available conversation datasets.

### Terms of use

Mistral-7B is released under the Apache 2.0 license

### References(s):

* Mistral 7B Instruct [Model Card](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) on Hugging Face
* Mistral 7B [paper](https://arxiv.org/abs/2310.06825) 
* Mistral 7B [blogpost](https://mistral.ai/news/announcing-mistral-7b/) 

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** Mistral-7B 
**Model version:** N/A 

### Input:

**Input Format:** Text 
**Input Parameters:** None 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** Text 
**Output Parameters:** None 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** N/A 
**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

## Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

mistral-7b-instruct

Mistral-7B-Instruct is a language model that can follow instructions, complete requests, and generate creative text formats.

## Model Overview

### Description:

Nemotron-3-8B-QA is a 8 billion parameter generative language model based on the Nemotron-3-8B base model. The model has been further fine-tuned for instruction following using Supervised Fine-tuning (SFT) using a method by NVIDIA specifically for Question and Answer method, to give concise and reliable answers.
Nemotron-3-8B-QA is part of NV-GPT, a family of enterprise ready decoder-only generative text models compatible with [NeMo Framework](https://www.nvidia.com/en-us/ai-data-science/generative-ai/nemo-framework).
NVIDIA NeMo is an end-to-end, cloud-native framework to build, customize, and deploy generative AI models anywhere. It includes training and inferencing frameworks, guardrailing toolkits, data curation tools, and pretrained models, offering enterprises an easy, cost-effective, and fast way to adopt generative AI.

### Terms of use

The use of this model is governed by the NVIDIA NeMo Foundational Models Community License Agreement

### References(s):

* Announcement Blog coming soon

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** GPT-3 
**Model version:** N/A 

### Input:

**Input Format:** Text 
**Input Parameters:** Temperature, TopP, TopK 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** Text 
**Output Parameters:** Max output tokens 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** NVIDIA AI Enterprise 
**Toolkit:** NeMo Framework 
**Supported Hardware Platform(s):** Hopper, Ampere 
**Supported Operating System(s):** Linux 

## Training & Finetuning:

### Dataset:

NVIDIA models are trained on a diverse set of public and proprietary datasets. This model was trained on a dataset containing 3.5 Trillion tokens of text. The dataset contains 53 different human languages and 37 programming languages. NVIDIA is committed to the responsible development of large language models and conducts reviews of all datasets included in training.

### Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

nemo-8b-qa

Nemotron-3-8B-QA is a 8 billion parameter generative language model based on the Nemotron-3-8B base model. The model has been further fine-tuned for instruction following by NVIDIA specifically for Question Answering.

## Model Overview

### Description:

Code Llama is a large language AI model built from a collection of models capable of generating code in response to prompts. It is based on [Llama 2](https://ai.meta.com/llama/).

### Terms of use

By accessing this model, you are agreeing to the Llama 2 Terms and Conditions of the [License](https://ai.meta.com/resources/models-and-libraries/llama-downloads), [Acceptable Use Policy](https://l.facebook.com/l.php?u=https%3A%2F%2Fai.meta.com%2Fllama%2Fuse-policy&h=AT17YXIWa1dxwuzDq96w614_oshaUCWIQYHHHf9BgC0X5UK39DW2tpSP6NzkeadQVs0XBaNlFSo4nydTUcKB5r-Kn4OLs_5IbMEZYAf5hUvyzIwx8-mLom2ic1vw5rSPFUoc9A) and [Meta’s Privacy Policy](https://www.facebook.com/privacy/policy/)

### References(s):

* [Code Llama: Open Foundation Models for Code](https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/) paper 
* [Meta's Code Llama](https://github.com/facebookresearch/codellama/blob/main/MODEL_CARD.md) model card 

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** Llama 2 

### Input:

**Input Format:** Text 
**Input Parameters:** Temperature, Top P (Nucleus Sampling) 

### Output:

**Output Format:** Text (code) 
**Output Parameters:** Max Output Tokens 

### Software Integration:

**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

## Training & Finetuning:

### Dataset:

Near-deduplicated dataset of publicly available code, propriety of Meta.
8% of samples data are sourced from natural language datasets related to code
and containing discussions about code and code snippets included in natural language questions or answers.
To help the model retain natural language understanding skills, dataset also inclues a small
proportion of batches from a natural language dataset.

## Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

codellama-34b

Code Llama is an LLM capable of generating code, and natural language about code, from both code and natural language prompts.

Code Generation

## Model Overview

### Description

The NVIDIA Retrieval QA Embedding Model is an embedding model optimized for text question-answering retrieval. An embedding model is a crucial component of a text retrieval system, as it transforms textual information into dense vector representations. They are typically transformer decoders that process tokens of input text (for example, question, passage) to output an embedding.

NVIDIA Retrieval QA Embedding Model is a part of NVIDIA NeMo Retriever, which provides state-of-the-art, commercially-ready models and microservices, optimized for the lowest latency and highest throughput. It features a production-ready information retrieval pipeline with enterprise support. The models that form the core of this solution have been trained using responsibly selected, auditable data sources. With multiple pre-trained models available as starting points, developers can also readily customize them for their domain-specific use cases, such as Information Technology, Human Resource help assistants, and Research & Development research assistants.

### Terms of use

The use of this model is governed by
the [NVIDIA AI Foundation Models Community License Agreement](https://developer.nvidia.com/downloads/nv-ai-foundation-models-license)

### References(s)

[Technical Blog](https://developer.nvidia.com/blog/build-enterprise-retrieval-augmented-generation-apps-with-nvidia-retrieval-qa-embedding-model/)

### Model Architecture

**Architecture Type:** Transformer 
**Network Architecture:** Fine-tuned E5-Large-Unsupervised retriever 

The NVIDIA Retrieval QA Embedding Model is a transformer encoder - a finetuned version of [E5-Large-Unsupervised](https://huggingface.co/intfloat/e5-large-unsupervised), with 24 layers and an embedding size of 1024, which is trained on private and public datasets as described in the Dataset and Training section. It supports a maximum input of 512 tokens.

Embedding models for text retrieval are typically trained using a bi-encoder architecture. This involves encoding a pair of sentences (for example, query and chunked passages) independently using the embedding model. Contrastive learning is used to maximize the similarity between the query and the passage that contains the answer, while minimizing the similarity between the query and sampled negative passages not useful to answer the question.

### Input

**Input Type:** text 
**Input Format:** list of strings 

### Output

**Output Type:** floats 
**Output Format:** list of float arrays, each array containing the embeddings for the corresponding input string. 

### Model Version(s)

NVIDIA Retrieval QA Embedding Model-1.0

## Training Dataset & Evaluation

### Training Dataset

The development of large-scale public open-QA datasets has enabled tremendous progress in powerful embedding models. However, one popular dataset named [MSMARCO](https://microsoft.github.io/msmarco/) restricts ‌commercial licensing, limiting the use of these models in commercial settings. To address this, we created our own internal open-domain QA dataset to train a commercially-viable embedding model. For NVIDIA proprietary data collection, we searched the passages from web logs and selected a collection of passages relevant to customer use cases for annotation by the NVIDIA internal data annotation team.

To minimize the redundancy in our data collection process, we selected samples that maximized relevancy distance scores and increased diversity in the data. The pretrained [E5-Large-Unsupervised](https://huggingface.co/intfloat/e5-large-unsupervised) embedding model was fine-tuned with contrastive learning with the prefix of “query:” for questions and “passage:” for context passages. Specifically, a mixture of English language datasets are used including our proprietary dataset, along with selected samples from commercially-viable public datasets. The AdamW optimizer is employed, incorporating 300 warm-up steps and 1e-6 learning rate with linear annealing schedule.

The training dataset details are as follows:

**Use Case**: Information retrieval for question and answering over text documents. 

**Data Sources**: 
- Public datasets licensed for commercial use. 
- Text from public websites. 
- Annotations created by NVIDIA’s internal team. 

**Language**: English (US) 

**Domains**: Knowledge, Description, Numeric (unit, time), Entity, Location, Person 

**Volume**: 40k internal proprietary samples, 200k samples from public dataset 

**High Level Schema**: 
- query: question text 
- doc: full document that contains the answer 
- chunk: section of the document that contains the answer 
- relevancy label: rating of how relevant the passage is to the question 
- span: exact token range in the chunk that contains the answer 

### Evaluation Results

We evaluated the NVIDIA Retrieval QA Embedding Model in comparison to literature open & commercial retriever models on academic benchmarks - [NQ](https://huggingface.co/datasets/BeIR/nq), [HotpotQA](https://huggingface.co/datasets/hotpot_qa) and [FiQA(Finance Q&A)](https://huggingface.co/datasets/BeIR/fiqa) from BeIR benchmark, and the [TechQA(Tech Support Q&A)](https://arxiv.org/pdf/1911.02984v1.pdf) dataset. In this benchmark, the metric used was [Normalized Discounted Cumulative Gain@10 (NDCG@10)](https://en.wikipedia.org/wiki/Discounted_cumulative_gain).

| Open & Commercial Retrieval Models | Average NDCG@10 on NQ, HotpotQA, FiQA, TechQA dataset|
|-----------------------------|-----------------------------|
| NVIDIA Retrieval QA | 53.86 |
| E5-Large_unsupervised | 45.70 |
| BM25 | 38.93 |

We also evaluated our embedding model with real internal customer datasets from telco, IT, consulting, and energy industries. The metric was Recall@5, to emulate a retrieval augmented generation (RAG) scenario where we would provide the top five most relevant passages as context in the prompt for the LLM model that is going to respond to the question. We compared our model’s information retrieval accuracy to a number of well-known embedding models made available by the AI community, including ones trained on non-commercial dataset (which are marked with "*").

| Retreval Model | Average Recall@5 on Internal Customer Datasets |
|-----------------------------|-----------------------------|
| NVIDIA Retrieval QA | 74.3% |
| DRAGON* | 72.7% |
| E5-Large* | 71.7% |
| BGE* | 71.1% |
| GTR* | 71.0% |
| Contriever* | 69.0% |
| GTE* | 63.9% |
| E5-Large_unsupervised | 61.6% |
| BM25 | 55.6% |

## Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards [here](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/nvolve-29k/bias). Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

### Intended use

The NVIDIA Retrieval QA Embedding model is most suitable for users who want to build a question and answer application over a large text corpus, leveraging the latest dense retrieval technologies.

**Ethical use**: Technology can have a profound impact on people and the world, and NVIDIA is committed to enabling trust and transparency in AI development. NVIDIA encourages users to adopt principles of AI ethics and trustworthiness to guide your business decisions by following the guidelines in the NVIDIA AI Foundation Models Community License Agreement.

### Limitations

The model was trained on the data that may contain toxic language and societal biases originally crawled from the Internet. Therefore, the model may amplify those biases, for example, associating certain genders with certain social stereotypes.

nvolve-29k

NVIDIA Retrieval QA Embedding is an embedding model that represents words, phrases, or other entities as vectors of numbers and understands the relation between words and phrases.

Embedding Model

codellama-13b

## Model Overview

### Description:

A genome-scale language foundation model (GenSLM) is an LLM trained on all known genomes from a virus or bacteria. It learns the evolutionary landscape of viruses like SARS-CoV-2 and can accurately and rapidly identify new variants.

This demo shows the visualization of the sequence log-likelihood (blue bar) and the cross-protein attention (orange lines), generated from the model, for a selected lineage to understand its co-evolutionary patterns.

### Terms of use

By accessing this model, you are agreeing to the [license](https://github.com/ramanathanlab/genslm#license), [acceptable use policy](https://github.com/ramanathanlab/genslm#citations)

### References(s):

* [Repo](https://github.com/ramanathanlab/genslm) 
* [Article](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9709791/) 

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** GenSLM

genslm

A genome-scale language foundation model (GenSLM) is an LLM trained on all known genomes from a virus or bacteria. It learns the evolutionary landscape of viruses like SARS-CoV-2 and can accurately and rapidly identify new variants.

Transformer Model

## Model Overview

### Description:

NeVA is NVIDIA's version of the LLaVA model where the open source LLaMA model is replaced with a GPT model trained by NVIDIA.
At a high level the image is encoded using a frozen hugging face CLIP model and projected to the text embedding dimensions. This is then concatenated with the embeddings of the prompt and passed in through the language model. Training happens in two stages:
* Pretraining: Here the language model is frozen and only the projection layer (that maps the image encoding to the embedding space) is trained. Here, image-caption pairs are used to pretrain the model.
* Finetuning: Here the language model is also trained along with the projection layer. To finetune the model synthetic instruction data generated using GPT4 is used.

### References(s):

* [Visual Instruction Tuning](https://arxiv.org/pdf/2304.08485.pdf) paper 
* [Blog](https://llava-vl.github.io) 
* [Codebase](https://github.com/haotian-liu/LLaVA) 
* [Demo](https://llava.hliu.cc) 

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** GPT + CLIP 
**Model version:** 8B, 22B, 43B 

### Input:

**Input Format:** Red, Green, Blue (RGB) Image + Text 
**Input Parameters:** temperature, max output tokens, quality, toxicity, humor, creativity, violence, helpfulness, not_appropriate 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** Text 
**Output Parameters:** None 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** N/A 
**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

## Training & Finetuning:

### Pretraining Dataset:

**Link:** [CC-3M](https://huggingface.co/datasets/liuhaotian/LLaVA-CC3M-Pretrain-595K) 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
The dataset consists of CC3M images and captions filtered to 595,000 samples. 

**Dataset License:**
* [COCO](https://cocodataset.org/#termsofuse)
* [CC-3M](https://github.com/google-research-datasets/conceptual-captions/blob/master/LICENSE)
* [BLIP](https://github.com/salesforce/BLIP/blob/main/LICENSE.txt) 

### Finetuning Dataset:

**Link:** [Synthetic data generated by GPT4](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K) 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
The data has 158,000 samples was generated synthetically by GPT4. It consists of a mix of short question answers, detailed image description, and higher level reasoning questions. 

**Dataset License:** [CC-BY-NC 4.0 License](https://creativecommons.org/licenses/by-nc/4.0/) CC BY-NC 4.0 

## Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

neva-22b

NeVA is a multi-modal vision-language model that understands text and images and generates informative responses.

llama2-13b

## Model Overview

### Description:

VISTA-3D is a specialized interactive foundation model for segmenting and anotating human anatomies.

VISTA-3D has three (3) core workflows:

- **Segment everything**: Enables whole body exploration, crucial for understanding complex diseases affecting multiple organs and for holistic treatment planning.
- **Segment using class**: Provides detailed sectional views based on specific classes, essential for targeted disease analysis or organ mapping, such as tumor identification in critical organs.
- **Segment point prompts**: Enhances segmentation precision through user-directed, click-based selection. This interactive approach accelerates the creation of accurate ground-truth data, essential in medical imaging analysis.

### Terms of use

By accessing this model, you are agreeing to the [license](https://developer.download.nvidia.com/assets/Clara/NVIDIA_MONAI_Service_Agreement_11_Oct_2023.pdf)

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** Custom, Multi-Head Structure 

### Input:

**Input Type(s):** CT Image 
**Input Format(s):** NIfTI (Neuroimaging Informatics Technology Initiative) 
**Input Parameters:** Three Dimensional (3D) 
**Other Properties Related to Input:** Array of Class/Point Information

### Output:

**Output Type(s):** Image 
**Output Format:** NIfTI 
**Output Parameters:** 3D 

### Software Integration:

**Supported Hardware Microarchitecture Compatibility:** Ampere, Hopper 
**Supported Operating System(s):** Linux 

### Inference:

**Engine:** Triton 

### Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

vista-3d

The Versatile Imaging SegmenTation and Annotation (VISTA) model combines semantic segmentation with interactivity, offering high accuracy and adaptability across diverse anatomical areas for medical imaging.

Image Segmentation

## Model Overview

### Description:

CLIP (Contrastive Language–Image Pretraining) is a model developed by OpenAI that connects vision and language in a way that enables it to understand and generate a wide array of image and text pairs. It has been trained on a large number of internet text–image pairs and can be applied to a variety of downstream tasks. 

### Terms of use

By accessing this model, you are agreeing to the CLIP terms and conditions of the [license](https://github.com/openai/CLIP/blob/main/LICENSE) and [acceptable use policy](https://github.com/openai/CLIP/blob/main/model-card.md#intended-use)

### References(s):

* [CLIP: Connecting Text and Images](https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language_Supervision.pdf) paper 
* [OpenAI's CLIP Model Card](https://github.com/openai/CLIP/blob/main/model-card.md) webpage 

### Model Architecture:

**Architecture Type:** Transformer for text and ViT/ResNet for images 
**Network Architecture:** CLIP 

### Input:

**Input Format:** Text and Red, Green, Blue (RGB) Image and Text 
**Input Parameters:** 2D 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** A number indicating the similarity score between image and text 
**Output Parameters:** 2D 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** Not Applicable (N/A) 
**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

### Training and Evaluation Dataset:

**Links:**
* [COYO-700M](https://github.com/kakaobrain/coyo-dataset) dataset 
* [LAION-Aesthetics](https://laion.ai/blog/laion-aesthetics) dataset 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
COYO-700M contains 747M image-text pairs. LAION-Aesthetics includes several collections of subsets from LAION 5B with high visual quality. 

**Dataset License:**
* [CC-BY-4.0 License](https://github.com/kakaobrain/coyo-dataset/blob/main/LICENSE.cc-by-4.0) for COYO-700M 
* License for LAION-Aesthetics not specified. 

## Inference:

**Engine:** [TensorRT](https://developer.nvidia.com/tensorrt), [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other (Not Listed)

clip

The CLIP (Contrastive Language-Image Pretraining) model combines vision and language using contrastive learning. It understands images and text together, enabling tasks like image classification and object detection.