Starcoder2-7B Overview

Description:

StarCoder2-7B generates source code from natural language instructions and code prompts across a wide range of programming languages. This 7-billion parameter model is part of the next generation of open-source large language models for code, developed by the BigCode collaboration, and was trained on The Stack v2, a massive, permissively licensed dataset covering over 600 programming languages. It is specifically designed to assist with tasks like code completion, code synthesis, and infilling (filling in missing code within a file).

This model is ready for commercial/non-commercial use.

This version introduces support for GB200 NVL72, GH200 NVL2, B200 and NVFP4. CUDA updated to version 12.9. For detailed information, refer to Release Notes for NVIDIA NIM for LLMs LLM 1.12.

Third-Party Community Consideration

This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party's requirements for this application and use case; see link to Non-NVIDIA [bigcode/starcoder2-7b]
(bigcode/starcoder2-7b · Hugging Face).

License/Terms of Use:

GOVERNING TERMS: The use of this model is governed by the NVIDIA Open Model License Agreement.

ADDITIONAL INFORMATION: BigCode Model License Agreement.

Get Help

Enterprise Support

Get access to knowledge base articles and support cases or submit a ticket. You are responsible for ensuring that your use of NVIDIA provided models complies with all applicable laws.

Deployment Geography:

Global

Use Case:

The expected users of StarCoder2-7B are software developers, data scientists, programmers, and students. They would use this model as a sophisticated coding assistant integrated into their development environment (like VS Code) or as a standalone tool for the following purposes:

• Accelerating Development: To significantly speed up the coding process using intelligent, context-aware code completion and infilling.

• Code Synthesis: To generate entire functions, classes, or scripts from a natural language description (e.g., "Write a JavaScript function that validates an email address using regex").

• Rapid Prototyping: To quickly create boilerplate code or functional prototypes for new applications or features.

• Learning & Debugging: To understand a new programming language or framework by generating example code, or to get suggestions on how to refactor or fix a piece of buggy code.

• Automation: To automate the creation of repetitive code, such as unit tests, data processing scripts, or configuration files.

Release Date:

Build.NVIDIA.com 03/18/2024 via
starcoder2-7b Model by BigCode | NVIDIA NIM

Github 02/28/2024 via
GitHub - bigcode-project/starcoder2: Home of StarCoder2!

Huggingface 02/28/2024 via
bigcode/starcoder2-7b · Hugging Face

Reference(s):

bigcode/starcoder2-7b · Hugging Face

Model Architecture:

Architecture Type: Transformer
Network Architecture: StarCoder2

This model was developed based on The Stack V2
https://huggingface.co/datasets/bigcode/the-stack-v2-train-full-ids

Number of model parameters: 0.717*10^10

Input:

Input Type(s): Text

Input Format: String

Input Parameters: One-Dimensional (1D)

Other Properties Related to Input:

• Tokens: The model accepts a sequence of tokens with a maximum context window of 16,384 tokens. This limit encompasses both the input prompt and the tokens generated by the model.
• Characters: Input strings can contain a wide range of Unicode characters, consistent with the diverse set of programming languages and natural languages found in the training data.
• Pre-Processing Needed: Raw input strings must be converted into a sequence of integer token IDs using the specific tokenizer associated with the StarCoder2-7B model before being fed into the network.

Output:

Output Type(s): Text

Output Format: String

Output Parameters: One-Dimensional (1D)

Other Properties Related to Output:

• Tokens: The model generates a sequence of tokens. The length of the output is variable and is ultimately constrained by the model's maximum context window of 16,384 tokens (the sum of input and output tokens cannot exceed this limit).
• Characters (Including Restrictions): The output consists of a wide range of Unicode characters from the model's vocabulary, designed to produce syntactically correct source code and coherent natural language.
• Post-Processing Needed: The raw token IDs generated by the model must be decoded using the model's specific tokenizer to be converted into a human-readable string. Special control tokens (e.g., end-of-sequence tokens) may also need to be filtered from the final output.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine: vLLM, TensorRT

Supported Hardware Microarchitecture Compatibility: NVIDIA Hopper

Preferred Operating System(s):

Linux
Windows

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

StarCoder2-7B-2.0.0

Usage

Running the model on CPU/GPU/multi GPU

• Using full precision:

# pip install git+https://github.com/huggingface/transformers.git # TODO: merge PR to main
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "bigcode/starcoder2-7b"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
# for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)

inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

• Using torch.bfloat16:

# pip install accelerate
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

checkpoint = "bigcode/starcoder2-7b"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

# for fp16 use `torch_dtype=torch.float16` instead
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", torch_dtype=torch.bfloat16)

inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to("cuda")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")

Quantized Versions through bitsandbytes

• Using 8-bit precision (int8):

# pip install bitsandbytes accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

# to use 4bit use `load_in_4bit=True` instead
quantization_config = BitsAndBytesConfig(load_in_8bit=True)

checkpoint = "bigcode/starcoder2-7b"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, quantization_config=quantization_config)

inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to("cuda")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
# load_in_8bit
Memory footprint: 7670.52 MB
# load_in_4bit
>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
Memory footprint: 4197.64 MB

Training, Testing, and Evaluation Datasets:

Training Dataset:

Data Modality: Text

Link: Undisclosed

Data Collection Method by dataset: Hybrid: Human, Synthetic, Automated

Labeling Method by dataset: Hybrid: Human, Synthetic, Automated

Properties:

• Quantity: StarCoder2-7B was trained on a processed subset of the dataset totaling approximately 3.3 trillion tokens. The complete, pre-processed Stack v2 dataset contains over 67TB of data.
• Dataset Descriptions: The Stack v2 is a very large dataset of source code from over 600 programming languages, supplemented with text from GitHub Issues, Pull Requests, and Kaggle Notebooks. A key feature of the dataset is that it has been filtered to only include content from repositories with permissive licenses. It also underwent an extensive PII (Personally Identifiable Information) redaction process to remove sensitive data.

Testing Dataset:

Link: Undisclosed

Data Collection Method by dataset: Hybrid: Human, Automated

Labeling Method by dataset: Hybrid: Human, Automated

Properties:

Quantity: The benchmarks consist of a varied number of problems (e.g., HumanEval: 164 problems; MBPP: ~400 test problems; DS-1000: 1,000 problems).

Dataset Descriptions: This is a collection of evaluation suites, not a single dataset. They are designed to measure the model's performance on specific downstream tasks:

• HumanEval/MBPP: Test Python code generation from natural language docstrings.
• DS-1000: Tests code generation for data science tasks (e.g., using Pandas, NumPy).
• CruxEval: Tests the model's ability to predict the output of a given code snippet. The primary goal is to assess the functional correctness and logical reasoning capabilities of the model.

Evaluation Dataset:

Link: Undisclosed

Data Collection Method by dataset: Hybrid: Automated, Human, Synthetic

Labeling Method by dataset: Hybrid: Human, Automated

Properties:

Quantity: The benchmarks consist of a varied number of problems (e.g., HumanEval: 164 problems; MBPP: ~400 test problems; DS-1000: 1,000 problems).

Dataset Descriptions: This is a collection of evaluation suites designed to measure the model's performance on specific downstream tasks:

• HumanEval/MBPP: Test Python code generation from natural language docstrings.
• DS-1000: Tests code generation for data science tasks (e.g., using Pandas, NumPy).
• CruxEval: Tests the model's ability to predict the output of a given code snippet. The primary goal is to assess the functional correctness and logical reasoning capabilities of the model.

Technical Limitations

The model has been trained on source code from 17 programming languages. The predominant language in source is English although other languages are also present. As such the model is capable of generating code snippets provided some context but the generated code is not guaranteed to work as intended. It can be inefficient and contain bugs or exploits. See the paper for an in-depth discussion of the model limitations.

Inference:

Acceleration Engine: vLLM, TensorRT

Test Hardware:

H100 SXM
H200 SXM (BF16 TP2)

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please report security vulnerabilities or NVIDIA AI Concerns here.

You are responsible for ensuring that your use of NVIDIA provided models complies with all applicable laws.