## Model Overview

### Description:

NeVA is NVIDIA's version of the LLaVA model where the open source LLaMA model is replaced with a GPT model trained by NVIDIA.
At a high level the image is encoded using a frozen hugging face CLIP model and projected to the text embedding dimensions. This is then concatenated with the embeddings of the prompt and passed in through the language model. Training happens in two stages:
* Pretraining: Here the language model is frozen and only the projection layer (that maps the image encoding to the embedding space) is trained. Here, image-caption pairs are used to pretrain the model.
* Finetuning: Here the language model is also trained along with the projection layer. To finetune the model synthetic instruction data generated using GPT4 is used.

### References(s):

* [Visual Instruction Tuning](https://arxiv.org/pdf/2304.08485.pdf) paper 
* [Blog](https://llava-vl.github.io) 
* [Codebase](https://github.com/haotian-liu/LLaVA) 
* [Demo](https://llava.hliu.cc) 

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** GPT + CLIP 
**Model version:** 8B, 22B, 43B 

### Input:

**Input Format:** Red, Green, Blue (RGB) Image + Text 
**Input Parameters:** temperature, max output tokens, quality, toxicity, humor, creativity, violence, helpfulness, not_appropriate 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** Text 
**Output Parameters:** None 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** N/A 
**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

## Training & Finetuning:

### Pretraining Dataset:

**Link:** [CC-3M](https://huggingface.co/datasets/liuhaotian/LLaVA-CC3M-Pretrain-595K) 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
The dataset consists of CC3M images and captions filtered to 595,000 samples. 

**Dataset License:**
* [COCO](https://cocodataset.org/#termsofuse)
* [CC-3M](https://github.com/google-research-datasets/conceptual-captions/blob/master/LICENSE)
* [BLIP](https://github.com/salesforce/BLIP/blob/main/LICENSE.txt) 

### Finetuning Dataset:

**Link:** [Synthetic data generated by GPT4](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K) 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
The data has 158,000 samples was generated synthetically by GPT4. It consists of a mix of short question answers, detailed image description, and higher level reasoning questions. 

**Dataset License:** [CC-BY-NC 4.0 License](https://creativecommons.org/licenses/by-nc/4.0/) CC BY-NC 4.0 

## Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

neva-22b

NeVA is a multi-modal vision-language model that understands text and images and generates informative responses.

NeVA-22B

Description

Publisher

Modified