Llama-3.3-70B-Instruct Overview

Description:

Llama-3.3-70B-Instruct is an auto-regressive language model that uses an optimized transformer architecture. It is designed for text-based tasks such as multilingual chat, coding assistance, and synthetic data generation, and is particularly optimized for dialogue-based use cases. With 70 billion parameters, it provides strong performance that is comparable to larger models but with lower hardware requirements, and it does not process images or audio.

This model is ready for commercial/non-commercial use.

Third-Party Community Consideration

This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party's requirements for this application and use case; see (meta-llama/Llama-3.3-70B-Instruct · Hugging Face).

License/Terms of Use:

GOVERNING TERMS: Use of this model is governed by the NVIDIA Open Model License Agreement.

ADDITIONAL INFORMATION: Llama 3.3 Community License Agreement. Built with Llama.

Get Help

Enterprise Support

Get access to knowledge base articles and support cases or submit a ticket. You are responsible for ensuring that your use of NVIDIA provided models complies with all applicable laws.

Deployment Geography:

Global

Use Case:

This model is intended for developers, researchers, and enterprises. They would integrate it into applications and workflows for a variety of advanced text-based tasks.

• For Conversational AI: Building sophisticated and natural-sounding chatbots for customer service, multilingual virtual assistants, and interactive dialogue systems.
• For Software Development: Engineers might use the model as a powerful coding assistant for generating code, debugging, explaining complex algorithms, and writing documentation.
• For Content Creation and Analysis: Businesses and content creators might use the model to draft emails, generate marketing copy, summarize long documents, and create synthetic text data to train other machine learning models.

Release Date:

Build.NVIDIA.com 12/17/2024 via
llama-3.3-70b-instruct Model by Meta | NVIDIA NIM

Github 12/13/2024 via
https://github.blog/changelog/2024-12-13-llama-3-3-70b-instruct-is-now-available-on-github-models-ga/

Huggingface 12/06/2024 via
https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct

Reference(s):

https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct

Model Architecture:

Architecture Type: Transformer
Network Architecture: Llama-3.3-70B

This model was developed based on Meta-Llama-3.3-70B
https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct.

Number of model parameters: 7.06*10^10

Input:

Input Type(s): Text

Input Format(s): String

Input Parameters: One-Dimensional (1D)

Other Properties Related to Input: The model processes input as tokens. The maximum context length is 8,192 tokens. Input text strings must be pre-processed by the model's specific Tiktoken tokenizer before being fed into the model.

Output:

Output Type(s): Text

Output Format(s): String

Output Parameters: One-Dimensional (1D)

Other Properties Related to Output: The model generates text as a sequence of tokens. The length of the generated output can be controlled by inference parameters. The raw token output requires post-processing (de-tokenization) to be converted into a human-readable string.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine: vLLM, TensorRT

Supported Hardware Microarchitecture Compatibility:

NVIDIA Ampere
NVIDIA Blackwell
NVIDIA Hopper
NVIDIA Lovelace

Preferred Operating System(s):

Linux
Windows

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

Llama-3.3-70B-Instruct

Usage

Use with transformers

Starting with transformers >= 4.45.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function.

Make sure to update your transformers installation via pip install --upgrade transformers.

See the snippet below for usage with Transformers:

import transformers
import torch

model_id = "meta-llama/Llama-3.3-70B-Instruct"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

outputs = pipeline(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])

Tool use with transformers

LLaMA-3.3 supports multiple tool use formats. You can see a full guide to prompt formatting here.

Tool use is also supported through chat templates in Transformers. Here is a quick example showing a single simple tool:

# First, define a tool
def get_current_temperature(location: str) -> float:
    """
    Get the current temperature at a location.
    
    Args:
        location: The location to get the temperature for, in the format "City, Country"
    Returns:
        The current temperature at the specified location in the specified units, as a float.
    """
    return 22.  # A real function should probably actually get the temperature!

# Next, create a chat and apply the chat template
messages = [
  {"role": "system", "content": "You are a bot that responds to weather queries."},
  {"role": "user", "content": "Hey, what's the temperature in Paris right now?"}
]

inputs = tokenizer.apply_chat_template(messages, tools=[get_current_temperature], add_generation_prompt=True)

You can then generate text from this input as normal. If the model generates a tool call, you should add it to the chat like so:

tool_call = {"name": "get_current_temperature", "arguments": {"location": "Paris, France"}}
messages.append({"role": "assistant", "tool_calls": [{"type": "function", "function": tool_call}]})

and then call the tool and append the result, with the tool role, like so:

messages.append({"role": "tool", "name": "get_current_temperature", "content": "22.0"})

After that, you can generate() again to let the model use the tool result in the chat. Note that this was a very brief introduction to tool calling - for more information, see the LLaMA prompt format docs and the Transformers tool use documentation.

Use with bitsandbytes

The model checkpoints can be used in 8-bit and 4-bit for further memory optimisations using bitsandbytes and transformers

See the snippet below for usage:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-3.3-70B-Instruct"
quantization_config = BitsAndBytesConfig(load_in_8bit=True)

quantized_model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", torch_dtype=torch.bfloat16, quantization_config=quantization_config)

tokenizer = AutoTokenizer.from_pretrained(model_id)
input_text = "What are we having for dinner?"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

output = quantized_model.generate(**input_ids, max_new_tokens=10)

print(tokenizer.decode(output[0], skip_special_tokens=True))

To load in 4-bit simply pass load_in_4bit=True

Use with llama

Please, follow the instructions in the repository.

To download Original checkpoints, see the example command below leveraging huggingface-cli:

huggingface-cli download meta-llama/Llama-3.3-70B-Instruct --include "original/*" --local-dir Llama-3.3-70B-Instruct

Training, Testing, and Evaluation Datasets:

Training Dataset

Data Modality: Text

Link: Undisclosed

Data Collection Method: Hybrid: Human, Synthetic, Automated

Labeling Method: Hybrid: Human, Synthetic

Properties:

The pre-training dataset contains over 15 trillion (15T) tokens from a diverse mix of publicly available online sources. The fine-tuning dataset consists of prompts and preference-ranked responses designed to improve helpfulness and safety.

Testing Dataset

Link: Undisclosed

Data Collection Method: Hybrid: Human, Synthetic, Automated

Labeling Method: Hybrid: Human, Automated

Properties:

The public datasets cover a wide range of tasks including massive multitask language understanding (MMLU), problem-solving (GSM8K), and code generation (HumanEval). Meta's internal evaluation set contains over 2,000 prompts designed to test for safety and helpfulness across various potentially risky categories.

Evaluation Dataset

Link: Undisclosed

Data Collection Method: Hybrid: Automated, Human, Synthetic

Labeling Method: Hybrid: Human, Automated

Properties:

The public datasets are industry-standard benchmarks designed to evaluate diverse capabilities like general knowledge, reasoning, coding, and math. For example, MMLU tests multitask knowledge, HumanEval tests code generation, and GSM8K tests grade-school math word problems. Meta's private evaluation set contains over 2,000 prompts for assessing safety and helpfulness.

Detailed Performance:

Technical Limitations

Testing conducted to date has not covered, nor could it cover, all scenarios. For these reasons, as with all LLMs, the model's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying this model in any applications, developers should perform safety testing and tuning tailored to their specific applications. Please refer to available resources including the Responsible Use Guide, Trust and Safety solutions, and other resources to learn more about responsible development.

Inference:

Acceleration Engine: vLLM, TensorRT

Test Hardware:

• B200 SXM
• H200 SXM
• H100 SXM
• A100 SXM 80GB
• A100 SXM 40GB
• L40S PCIe
• A10G
• H100 NVL
• H200 NVL
• GH200 96GB
• GB200 NVL72
• GH200 NVL2
• RTX 5090
• RTX 4090
• RTX 6000 Ada

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please report security vulnerabilities or NVIDIA AI Concerns here.

You are responsible for ensuring that your use of NVIDIA provided models complies with all applicable laws.