Built with Llama - The Meta Llama 3.2 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction-tuned text-to-text generative models in 1B and 3B sizes.
Model Overview
Description:
Built with Llama - The Meta Llama 3.2 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction-tuned text-to-text generative models in 1B and 3B sizes . The Llama 3.2 instruction-tuned text only models are optimized for multilingual dialogue use cases, including agentic retrieval and summarization tasks. They outperform many of the available open source and closed chat models on common industry benchmarks.
This model is ready for commercial use.
Llama 3.2 3B Instruct INT4 ONNX
Llama 3.2 3B Instruct model is quantized to AWQ INT4 using AutoAWQ and converted to ONNX using Onnxruntime-GenAI.
Third-Party Community Consideration Llama 3.2
This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party’s requirements for this application and use case; see link to the Llama 3.2 Model Card.
License/Terms of Use:
This model is governed by the NVIDIA AI Foundation Models Community License Agreement . Additional Information Llama 3.2 Community License Agreement, Built with Llama Acceptable Use Policy
Reference(s):
Model Architecture:
Llama 3.2 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.
Architecture Type: Large Language Model (LLM)
Network Architecture: llama
Training, Testing, and Evaluation Datasets:
Refer to Llama 3.2 Model Card for the details.
Input:
Input Type: Text and Code
Input Format: String
Input Parameters: Temperature, TopP
Other Properties Related to Input: Supports English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai
Output:
Output Type(s): Text and Code
Output Format: String
Output Parameters: Max output tokens
Software Integration:
Runtime(s): Not Applicable
Supported Hardware Platform(s): RTX 4090, 6GB or higher VRAM gpus are recommended. Higher VRAM may be required for larger context length use cases.
Supported Operating System(s): Windows
Model Version(s): 1.0
Inference:
Inference Backend: Onnxruntime-GenAI-DirectML
(Note: please refer to ReadMe.txt for the detailed instructions.)
Accuracy Scores:
MMLU (5# shots): 58.58
Evaluation Dataset:
Link: https://people.eecs.berkeley.edu/~hendrycks/data.tar .
Data Collection Method by dataset = Unknown
Labeling Method by dataset = Not Applicable
Test configuration:
-
GPU: RTX 4090.
-
Windows 11: 23H2
-
NVIDIA Graphics driver: R560 or higher