## Model Overview

### Description:

Code Llama is a large language AI model built from a collection of models capable of generating code in response to prompts. It is based on [Llama 2](https://ai.meta.com/llama/).

### Terms of use

By accessing this model, you are agreeing to the LLama 2 Terms and Conditions of the [License](https://ai.meta.com/resources/models-and-libraries/llama-downloads), [Acceptable Use Policy](https://l.facebook.com/l.php?u=https%3A%2F%2Fai.meta.com%2Fllama%2Fuse-policy&h=AT17YXIWa1dxwuzDq96w614_oshaUCWIQYHHHf9BgC0X5UK39DW2tpSP6NzkeadQVs0XBaNlFSo4nydTUcKB5r-Kn4OLs_5IbMEZYAf5hUvyzIwx8-mLom2ic1vw5rSPFUoc9A) and [Meta’s Privacy Policy](https://www.facebook.com/privacy/policy/)

### References(s):

* [Code Llama: Open Foundation Models for Code](https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/) paper 
* [Meta's Code Llama](https://github.com/facebookresearch/codellama/blob/main/MODEL_CARD.md) model card 

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** Llama 2 

### Input:

**Input Format:** Text 
**Input Parameters:** Temperature, Top P (Nucleus Sampling) 

### Output:

**Output Format:** Text (code) 
**Output Parameters:** Max Output Tokens 

### Software Integration:

**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

## Training & Finetuning:

### Dataset:

Near-deduplicated dataset of publicly available code, propriety of Meta.
8% of samples data are sourced from natural language datasets related to code
and containing discussions about code and code snippets included in natural language questions or answers.
To help the model retain natural language understanding skills, dataset also inclues a small
proportion of batches from a natural language dataset.

### Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

codellama

Code Llama is a code-specialized version of Llama 2. It can generate code, and natural language about code, from both code and natural language prompts.

Code Generation

## Model Overview

### Description:

NeVA is NVIDIA's version of the LLaVA model where the open source LLaMA model is replaced with a GPT model trained by NVIDIA.
At a high level the image is encoded using a frozen hugging face CLIP model and projected to the text embedding dimensions. This is then concatenated with the embeddings of the prompt and passed in through the language model. Training happens in two stages:
* Pretraining: Here the language model is frozen and only the projection layer (that maps the image encoding to the embedding space) is trained. Here, image-caption pairs are used to pretrain the model.
* Finetuning: Here the language model is also trained along with the projection layer. To finetune the model synthetic instruction data generated using GPT4 is used.

### References(s):

* [Visual Instruction Tuning](https://arxiv.org/pdf/2304.08485.pdf) paper 
* [Blog](https://llava-vl.github.io) 
* [Codebase](https://github.com/haotian-liu/LLaVA) 
* [Demo](https://llava.hliu.cc) 

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** GPT + CLIP 
**Model version:** 8B, 22B, 43B 

### Input:

**Input Format:** Red, Green, Blue (RGB) Image + Text 
**Input Parameters:** temperature, max output tokens, quality, toxicity, humor, creativity, violence, helpfulness, not_appropriate 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** Text 
**Output Parameters:** None 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** N/A 
**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

## Training & Finetuning:

### Pretraining Dataset:

**Link:** [CC-3M](https://huggingface.co/datasets/liuhaotian/LLaVA-CC3M-Pretrain-595K) 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
The dataset consists of CC3M images and captions filtered to 595,000 samples. 

**Dataset License:**
* [COCO](https://cocodataset.org/#termsofuse)
* [CC-3M](https://github.com/google-research-datasets/conceptual-captions/blob/master/LICENSE)
* [BLIP](https://github.com/salesforce/BLIP/blob/main/LICENSE.txt) 

### Finetuning Dataset:

**Link:** [Synthetic data generated by GPT4](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K) 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
The data has 158,000 samples was generated synthetically by GPT4. It consists of a mix of short question answers, detailed image description, and higher level reasoning questions. 

**Dataset License:** [CC-BY-NC 4.0 License](https://creativecommons.org/licenses/by-nc/4.0/) CC BY-NC 4.0 

### Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

neva

NeVA is a multi-modal vision-language model that understands text and images and generates informative responses.

Image Conversation

## Model Overview

### Description:

SDXL is a latent diffusion model for text-to-image synthesis. Compared to previous versions of Stable Diffusion, SDXL leverages a three times larger UNet backbone: The increase of model parameters is mainly due to more attention blocks and a larger cross-attention context as SDXL uses a second text encoder.

### Terms of use

By accessing this model, you are agreeing to the SDXL 1.0 terms and conditions of the [license](https://github.com/Stability-AI/generative-models/blob/main/model_licenses/LICENSE-SDXL1.0), [acceptable use policy](https://github.com/Stability-AI/stablediffusion/blob/main/LICENSE) and [stability.ai privacy policy](https://platform.stability.ai/legal/privacy-policy)

### References(s):

* [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://arxiv.org/abs/2307.01952) paper 
* [Stability-AI](https://github.com/Stability-AI/generative-models) repo 
* [Stability-AI's SDXL Model Card](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) webpage 

### Model Architecture:

**Architecture Type:** Transformer and Convolutional Neural Network (CNN) 
**Network Architecture:** UNet + attention blocks 
**Model version:** SDXL 1.0 

### Input:

**Input Format:** Text 
**Input Parameters:** scheduler type, denoising steps, classifier-free guidance 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** Red, Green, Blue (RGB) Image 
**Output Parameters:** 2D 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** N/A 
**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

## Training & Finetuning:

### Dataset:

**Link:** [LAION 5B](https://laion.ai/blog/laion-5b/) 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
The dataset consists of 5.85 billion CLIP-filtered image-text pairs. 

**Dataset License:**
* [CreativeML Open RAIL-M license](https://huggingface.co/spaces/CompVis/stable-diffusion-license) 

### Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

sdxl

Stable Diffusion XL (SDXL) enables you to generate expressive images with shorter prompts and insert words inside images.

Image Generation

## Model Overview

### Description:

Llama 2 is a large language AI model comprising a collection of models capable of generating text and code in response to prompts.

### Terms of use

By accessing this model, you are agreeing to the LLama 2 terms and conditions of the [license](https://l.facebook.com/l.php?u=https%3A%2F%2Fai.meta.com%2Fllama%2Flicense&h=AT0DpDKXSE-Yi_pulcu4-5oKNHyiayGrLbQqW7qI_4Ez5y-67Nmyyu38D5NcEmGSifLOtDZAJaQOPMGJg8zCFDJjbYcyJZQPawxWU3-GaVdgStbboYT00yqZvyrxP1DVlbfAog), [acceptable use policy](https://l.facebook.com/l.php?u=https%3A%2F%2Fai.meta.com%2Fllama%2Fuse-policy&h=AT17YXIWa1dxwuzDq96w614_oshaUCWIQYHHHf9BgC0X5UK39DW2tpSP6NzkeadQVs0XBaNlFSo4nydTUcKB5r-Kn4OLs_5IbMEZYAf5hUvyzIwx8-mLom2ic1vw5rSPFUoc9A) and [Meta’s privacy policy](https://www.facebook.com/privacy/policy/)

### References(s):

* [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://scontent-lhr8-2.xx.fbcdn.net/v/t39.2365-6/10000000_662098952474184_2584067087619170692_n.pdf?_nc_cat=105&ccb=1-7&_nc_sid=3c67a6&_nc_ohc=SvLw337Y0sQAX8zTcqT&_nc_ht=scontent-lhr8-2.xx&oh=00_AfC2GBa-ZPdVLSxOnnjuTZHTDu51NZUcR6KubtLe5tEUbw&oe=64D0333F) paper 
* [Meta's Llama 2](https://ai.meta.com/llama/) webpage 
* [Meta's Llama 2 Model Card](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md) webpage 

### Model Architecture:

**Architecture Type:** Transformer 
**Network Architecture:** Llama 2 
**Model version:** N/A 

### Input:

**Input Format:** Text 
**Input Parameters:** Temperature, TopP 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** Text 
**Output Parameters:** Max output tokens 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** N/A 
**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

## Training & Finetuning:

### Dataset:

**Link:**
* [Falcon RefinedWeb](https://arxiv.org/abs/2306.01116)
* [BigCode - The Stack (StarCoder)](https://huggingface.co/datasets/bigcode/the-stack)
* [Wikipedia](https://www.tensorflow.org/datasets/catalog/wikipedia)
* [ArXiv](https://arxiv.org/pdf/2101.00027.pdf)
* [Books3](https://arxiv.org/pdf/2101.00027.pdf)
* [StackExchange](https://archive.org/details/stackexchange) 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
* Falcon RefinedWeb is a massive English web dataset containing 500-650GT depending on the used tokenizer
* The Stack dataset is a collection of source code in over 300 programming languages
* Wikipedia is 20.54GB of online encyclopedia written and maintained by a community of volunteers
* The Pile: ArXiv is 56.21GB of ArXiv papers contents
* The Pile: Books3 is 100.96GB dataset of books derived from a copy of the contents of the Bibliotik private tracker made available by Shawn Presser
* StackExchange is an anonymized dump of all user-contributed content on the Stack Exchange network where each site is formatted as a separate archive consisting of zipped XML files

**Dataset License:**
Free for research and commercial use.

### Inference:

**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other

llama2

Llama 2 is a large language AI model capable of generating text and code in response to prompts.

Text Generation

## Model Overview

### Description:

CLIP (Contrastive Language–Image Pretraining) is a model developed by OpenAI that connects vision and language in a way that enables it to understand and generate a wide array of image and text pairs. It has been trained on a large number of internet text–image pairs and can be applied to a variety of downstream tasks. 

### Terms of use

By accessing this model, you are agreeing to the CLIP terms and conditions of the [license](https://github.com/openai/CLIP/blob/main/LICENSE) and [acceptable use policy](https://github.com/openai/CLIP/blob/main/model-card.md#intended-use)

### References(s):

* [CLIP: Connecting Text and Images](https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language_Supervision.pdf) paper 
* [OpenAI's CLIP Model Card](https://github.com/openai/CLIP/blob/main/model-card.md) webpage 

### Model Architecture:

**Architecture Type:** Transformer for text and ViT/ResNet for images 
**Network Architecture:** CLIP 

### Input:

**Input Format:** Text and Red, Green, Blue (RGB) Image and Text 
**Input Parameters:** 2D 
**Other Properties Related to Output:** None 

### Output:

**Output Format:** A number indicating the similarity score between image and text 
**Output Parameters:** 2D 
**Other Properties Related to Output:** None 

### Software Integration:

**Runtime(s):** Not Applicable (N/A) 
**Supported Hardware Platform(s):** Hopper, Ampere/Turing 
**Supported Operating System(s):** Linux 

### Training and Evaluation Dataset:

**Links:**
* [COYO-700M](https://github.com/kakaobrain/coyo-dataset) dataset 
* [LAION-Aesthetics](https://laion.ai/blog/laion-aesthetics) dataset 

**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
COYO-700M contains 747M image-text pairs. LAION-Aesthetics includes several collections of subsets from LAION 5B with high visual quality. 

**Dataset License:**
* [CC-BY-4.0 License](https://github.com/kakaobrain/coyo-dataset/blob/main/LICENSE.cc-by-4.0) for COYO-700M 
* License for LAION-Aesthetics not specified. 

### Inference:

**Engine:** [TensorRT](https://developer.nvidia.com/tensorrt), [Triton](https://developer.nvidia.com/triton-inference-server) 
**Test Hardware:** Other (Not Listed)

clip

The CLIP (Contrastive Language-Image Pretraining) model combines vision and language using contrastive learning. It understands images and text together, enabling tasks like image classification and object detection.

Experience AI in Action