NVIDIA NeMo Evaluator

The goal of NVIDIA NeMo Evaluator is to advance and refine state-of-the-art methodologies for model evaluation, and deliver them as modular evaluation packages (evaluation containers and pip wheels) that teams can use as standardized building blocks.

Long Context Eval

Long Context Eval is part of NVIDIA NeMo Evaluator, providing standardized evaluation methodologies for assessing LLMs on long-context understanding tasks.

Overview

Long Context Eval evaluates models' ability to process and understand extended context windows, testing comprehension and reasoning across long documents.

Key Features:

• Evaluation of long-context understanding capabilities
• Support for extended context lengths
• Comprehensive benchmarking across various long-form tasks

Quick Start Guide

List the available evaluations:

$ nemo-evaluator ls
ruler:
  * ruler-128k-chat
  * ruler-128k-completions
  * ruler-16k-chat
  * ruler-16k-completions
  * ruler-1m-chat
  * ruler-1m-completions
  * ruler-256k-chat
  * ruler-256k-completions
  * ruler-32k-chat
  * ruler-32k-completions
  * ruler-4k-chat
  * ruler-4k-completions
  * ruler-512k-chat
  * ruler-512k-completions
  * ruler-64k-chat
  * ruler-64k-completions
  * ruler-8k-chat
  * ruler-8k-completions
  * ruler-chat
  * ruler-completions

NVIDIA NeMo Evaluator provides you with evaluation clients that are specifically built to evaluate model endpoints using our Standard API.

Launching an Evaluation

Run the Evaluation of Your Choice

export API_KEY=your_nvidia_api_key_here

nemo-evaluator run_eval \
    --eval_type ruler-64k-chat \
    --model_id meta/llama-3.1-70b-instruct \
    --model_type chat \
    --model_url https://integrate.api.nvidia.com/v1/chat/completions \
    --api_key_name API_KEY \
    --output_dir './results/long_context_eval'

3rd Party Source Code

Users can download the third party source code through the URL provided in the container's README located in workdir.