Llama-3.1-8B-Instruct Overview

Description:

Llama-3.1-8B-Instruct is an 8 billion parameter, instruction-tuned large language model created by Meta. This model is part of the Llama 3.1 family of open-access models and is specifically optimized for dialogue and conversational use cases, making it highly capable of following user instructions to perform a wide variety of natural language processing tasks.

This model is ready for commercial/non-commercial use.

Third-Party Community Consideration

This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party's requirements for this application and use case; see link to Non-NVIDIA [meta-llama/Llama-3.1-8B-Instruct]
(https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct).

License/Terms of Use:

GOVERNING TERMS: NVIDIA Open Model License.

ADDITIONAL INFORMATION: Llama 3.1 Community License Agreement. Built with Llama.

Get Help

Enterprise Support

Get access to knowledge base articles and support cases or submit a ticket. You are responsible for ensuring that your use of NVIDIA provided models complies with all applicable laws.

Deployment Geography:

Global

Use Case:

This model is primarily used by developers, researchers, and businesses to build and experiment with a wide range of generative AI applications. Its combination of strong performance, efficiency, and an open-access license makes it highly versatile.

• Developers and Businesses would use this model to create production-ready applications such as:
  • AI Chatbots and customer service agents.
  • Content creation tools for writing emails, marketing copy, and articles.
  • Summarization and question-answering systems for internal documents.
  • Code generation assistants to help programmers write and debug code.
• Researchers would use it to study large language model behavior, explore AI safety and alignment, and benchmark new training or fine-tuning techniques on a capable, open model.
• AI Hobbyists would use the model for personal projects, running it on consumer-grade hardware to experiment with creating their own AI assistants or exploring the frontiers of generative AI.

Release Date:

Build.NVIDIA.com 07/23/2024 via
https://build.nvidia.com/meta/llama-3_1-8b-instruct

Huggingface 07/23/2024 via
https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

Reference(s):

https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

Model Architecture:

Architecture Type: Transformer
Network Architecture: Llama-3.1-8B

This model was developed based on Meta Llama-3.1-8B
https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

Number of model parameters: 8.03*10^9

Input:

Input Type(s): Text

Input Format(s): String

Input Parameters: One-Dimensional (1D)

Other Properties Related to Input: The primary input limit is the model's maximum context length, which is 128,000 tokens. All input text must be pre-processed using the model's specific tokenizer to convert the string into a sequence of token IDs. For conversational use, inputs should be formatted using the model's designated chat template.

Output:

Output Type(s): Text

Output Format(s): String

Output Parameters: One-Dimensional (1D)

Other Properties Related to Output: The output consists of a variable-length sequence of tokens from the model's vocabulary. The total length of the input and output cannot exceed the 128,000-token context window. Post-processing is required to detokenize the raw token ID sequence into a human-readable string.

Software Integration:

Runtime Engine: vLLM, TensorRT

Supported Hardware Microarchitecture Compatibility:

NVIDIA Ampere
NVIDIA Blackwell
NVIDIA Hopper
NVIDIA Lovelace

Preferred Operating System(s):

Linux
Windows
macOS

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

Llama-3.1-8B-Instruct

Usage

Use with transformers

Starting with transformers >= 4.43.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function.

Make sure to update your transformers installation via pip install --upgrade transformers.

import transformers
import torch

model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

outputs = pipeline(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])

Note: You can also find detailed recipes on how to use the model locally, with torch.compile(), assisted generations, quantised and more at huggingface-llama-recipes

Tool use with transformers

LLaMA-3.1 supports multiple tool use formats. You can see a full guide to prompt formatting here.

Tool use is also supported through chat templates in Transformers. Here is a quick example showing a single simple tool:

# First, define a tool
def get_current_temperature(location: str) -> float:
    """
    Get the current temperature at a location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
    Returns:
        The current temperature at the specified location in the specified units, as a float.
    """
    return 22.  # A real function should probably actually get the temperature!

# Next, create a chat and apply the chat template
messages = [
  {"role": "system", "content": "You are a bot that responds to weather queries."},
  {"role": "user", "content": "Hey, what's the temperature in Paris right now?"}
]

inputs = tokenizer.apply_chat_template(messages, tools=[get_current_temperature], add_generation_prompt=True)

You can then generate text from this input as normal. If the model generates a tool call, you should add it to the chat like so:

tool_call = {"name": "get_current_temperature", "arguments": {"location": "Paris, France"}}
messages.append({"role": "assistant", "tool_calls": [{"type": "function", "function": tool_call}]})

and then call the tool and append the result, with the tool role, like so:

messages.append({"role": "tool", "name": "get_current_temperature", "content": "22.0"})

After that, you can generate() again to let the model use the tool result in the chat. Note that this was a very brief introduction to tool calling - for more information, see the LLaMA prompt format docs and the Transformers tool use documentation.

Training, Testing, and Evaluation Datasets:

Training Dataset

Data Modality: Text

Link: Undisclosed

Data Collection Method: Hybrid: Human, Synthetic, Automated

Labeling Method: Hybrid: Human, Automated

Properties:

The model was pre-trained on a dataset of over 15 trillion tokens. This dataset is a high-quality mix of publicly available data, heavily filtered for safety and quality. The instruction fine-tuning dataset is smaller and consists of high-quality prompts, responses, and preference rankings curated by humans.

Testing Dataset

Link: Undisclosed

Data Collection Method: Hybrid: Human, Automated

Labeling Method: Human

Properties:

The testing datasets comprise thousands of individual problems designed to measure model capabilities in specific areas:

• General Knowledge & Reasoning: MMLU, DROP, AGIEval, BIG-Bench Hard
• Mathematics: GSM8K, MATH
• Coding: HumanEval

Evaluation Dataset

Link: Undisclosed

Data Collection Method: Hybrid: Human, Automated

Labeling Method: Human

Properties:

The evaluation datasets consist of thousands of questions and problems designed to test the model's capabilities in areas like general knowledge, reasoning, mathematics, and coding.

Base pretrained models

Instruction tuned models

Multilingual benchmarks

Technical Limitations

Testing conducted to date has not covered, nor could it cover, all scenarios. For these reasons, as with all LLMs, the model's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying this model in any applications, developers should perform safety testing and tuning tailored to their specific applications. Please refer to available resources including the Responsible Use Guide, Trust and Safety solutions, and other resources to learn more about responsible development.

Inference:

Acceleration Engine: vLLM, TensorRT

Test Hardware:

DC P0:

• B200 SXM
  
• H200 SXM
  
• H100 SXM
  
• A100 SXM 40GB
  
• A100 SXM 80GB
  
• A10G
  
• L40S PCIe
  
• H100 NVL
  
• H200 NVL
  
• GB200 NVL72
  
• GH200 NVL2
  
• GH200 96GB
  
• RTX 6000 Pro BSE
  
• DGX Spark
  

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please report security vulnerabilities or NVIDIA AI Concerns here.

You are responsible for ensuring that your use of NVIDIA provided models complies with all applicable laws.