Llama-3.1-70B-Instruct Overview

Description:

Llama-3.1-70B-Instruct is a multilingual large language model from the Meta Llama 3.1 collection of pretrained and instruction-tuned generative models. This model is optimized for multilingual dialogue use cases and outperforms many available open-source and closed-source chat models on common industry benchmarks. Llama 3.1 is an auto-regressive language model that uses an optimized transformer architecture; the tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.

This model is ready for commercial/non-commercial use.

Third-Party Community Consideration

This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party's requirements for this application and use case; see link to meta-llama/Llama-3.1-70B-Instruct.

License/Terms of Use:

GOVERNING TERMS: Use of this model is governed by the NVIDIA Open Model License Agreement.

ADDITIONAL INFORMATION: Llama 3.1 Community License Agreement. Built with Llama.

Get Help

Enterprise Support

Get access to knowledge base articles and support cases or submit a ticket.
You are responsible for ensuring that your use of NVIDIA provided models complies with all applicable laws.

Deployment Geography:

Global

Use Case:

Developers, AI researchers, and businesses would be expected to use this system to build and power a wide range of applications that require advanced reasoning, instruction-following, and multilingual dialogue capabilities. Specific applications include creating sophisticated chatbots and virtual assistants, developing powerful content creation and summarization tools, building complex question-answering systems, and powering multilingual customer support platforms.

Release Date:

Build.NVIDIA.com 07/23/2024 via
llama-3.1-70b-instruct Model by Meta | NVIDIA NIM

Github 07/23/2024 via
https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE

Huggingface 07/23/2024 via
https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct

Reference(s):

https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct

Model Architecture:

Architecture Type: Transformer
Network Architecture: Llama-3.1-70B

This model was developed based on Meta-Llama-3.1-70B
https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct

Number of model parameters: 7.06*10^10

Input:

Input Type(s): Text

Input Format(s): String

Input Parameters: One-Dimensional (1D)

Other Properties Related to Input: The model accepts a string of text which is converted into tokens using the model's specific tokenizer. The total length of the input prompt and the generated output cannot exceed the model's context window of 128,000 tokens.

Output:

Output Type(s): Text

Output Format(s): String

Output Parameters: One-Dimensional (1D)

Other Properties Related to Output: The model generates a string of text, produced token by token. The maximum length of the output is limited by the model's 128,000-token context window, less the length of the input prompt. Post-processing is required to decode the model's token-based output into a readable string.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine: vLLM, TensorRT

Supported Hardware Microarchitecture Compatibility:

NVIDIA Ampere
NVIDIA Blackwell
NVIDIA Hopper
NVIDIA Lovelace

Preferred Operating System(s):

Linux
Windows

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

Llama-3.1-70B-Instruct

Usage

Use with transformers

Starting with transformers >= 4.45.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function.

Make sure to update your transformers installation via pip install --upgrade transformers.

See the snippet below for usage with Transformers:

import transformers
import torch

model_id = "meta-llama/Meta-Llama-3.1-70B-Instruct"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

outputs = pipeline(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])

Tool use with transformers

LLaMA-3.3 supports multiple tool use formats. You can see a full guide to prompt formatting here.

Tool use is also supported through chat templates in Transformers. Here is a quick example showing a single simple tool:

# First, define a tool
def get_current_temperature(location: str) -> float:
    """
    Get the current temperature at a location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
    Returns:
        The current temperature at the specified location in the specified units, as a float.
    """
    return 22.  # A real function should probably actually get the temperature!

# Next, create a chat and apply the chat template
messages = [
  {"role": "system", "content": "You are a bot that responds to weather queries."},
  {"role": "user", "content": "Hey, what's the temperature in Paris right now?"}
]

inputs = tokenizer.apply_chat_template(messages, tools=[get_current_temperature], add_generation_prompt=True)

You can then generate text from this input as normal. If the model generates a tool call, you should add it to the chat like so:

tool_call = {"name": "get_current_temperature", "arguments": {"location": "Paris, France"}}
messages.append({"role": "assistant", "tool_calls": [{"type": "function", "function": tool_call}]})

and then call the tool and append the result, with the tool role, like so:

messages.append({"role": "tool", "name": "get_current_temperature", "content": "22.0"})

After that, you can generate() again to let the model use the tool result in the chat. Note that this was a very brief introduction to tool calling - for more information, see the LLaMA prompt format docs and the Transformers tool use documentation.

Use with bitsandbytes

The model checkpoints can be used in 8-bit and 4-bit for further memory optimisations using bitsandbytes and transformers

See the snippet below for usage:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Meta-Llama-3.1-70B-Instruct"
quantization_config = BitsAndBytesConfig(load_in_8bit=True)

quantized_model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", torch_dtype=torch.bfloat16, quantization_config=quantization_config)

tokenizer = AutoTokenizer.from_pretrained(model_id)
input_text = "What are we having for dinner?"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

output = quantized_model.generate(**input_ids, max_new_tokens=10)

print(tokenizer.decode(output[0], skip_special_tokens=True))

To load in 4-bit simply pass load_in_4bit=True

Use with llama

Please, follow the instructions in the repository.

To download Original checkpoints, see the example command below leveraging huggingface-cli:

huggingface-cli download meta-llama/Meta-Llama-3.1-70B-Instruct --include "original/*" --local-dir Meta-Llama-3.1-70B-Instruct

Training, Testing, and Evaluation Datasets:

Training Dataset

Data Modality: Text

Link: Undisclosed

Data Collection Method: Hybrid: Human, Synthetic, Automated

Labeling Method: Hybrid: Automated, Human

Properties:

The pretraining dataset contains over 15 trillion tokens. It is a multilingual dataset covering over 30 languages and was filtered heavily for quality using various techniques, including heuristic filters, NSFW filters, and text classifiers. The model's knowledge was trained on data with a cutoff of December 2023.

Testing Dataset

Link: Undisclosed

Data Collection Method: Hybrid: Human, Synthetic, Automated

Labeling Method: Hybrid: Automated, Human

Properties:

Description:

The model was tested on a diverse set of evaluation data.

• Public Benchmarks: These test a wide range of capabilities, from general knowledge and reasoning (MMLU, HellaSwag) to expert-level problem-solving (GPQA) and programming (HumanEval, which contains 164 programming problems).
• Internal Evaluation Set: Meta created a new high-quality test set of 2,000 prompts covering 12 key use cases (e.g., coding, reasoning, creative writing, instruction following). This set is used for human evaluation to assess performance on real-world, nuanced tasks.

Evaluation Dataset

Link: Undisclosed

Data Collection Method: Hybrid: Automated, Human, Synthetic

Labeling Method: Hybrid: Human, Automated

Properties:

The evaluation datasets are diverse and test a wide spectrum of capabilities. MMLU measures broad multitask knowledge. GPQA assesses advanced reasoning with difficult, expert-level questions. HumanEval and MATH specifically test code generation and mathematical reasoning abilities, respectively. Meta also utilizes a large, private, human-annotated evaluation set designed to assess model performance in real-world, nuanced scenarios.

Base pretrained models

Instruction tuned models

Multilingual benchmarks

Technical Limitations

Testing conducted to date has not covered, nor could it cover, all scenarios. For these reasons, as with all LLMs, the model's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying this model in any applications, developers should perform safety testing and tuning tailored to their specific applications. Please refer to available resources including the Responsible Use Guide, Trust and Safety solutions, and other resources to learn more about responsible development.

Inference:

Acceleration Engine: vLLM, TensorRT

Test Hardware:

DC P0:

• B200 SXM
  
• H200 SXM
  
• H100 SXM
  
• A100 SXM 80GB
  
• A100 SXM 40GB
  
• L40S PCIe
  
• A10G
  
• H100 NVL
  
• H200 NVL
  
• GH200 96GB
  
• GB200 NVL72
  
• GH200 NVL2
  
• RTX 6000 Pro BSE
  

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please report security vulnerabilities or NVIDIA AI Concerns here.

You are responsible for ensuring that your use of NVIDIA provided models complies with all applicable laws.