NVIDIA NeMo Evaluator

The goal of NVIDIA NeMo Evaluator is to advance and refine state-of-the-art methodologies for model evaluation, and deliver them as modular evaluation packages (evaluation containers and pip wheels) that teams can use as standardized building blocks.

Overview

Tau2-Bench implements a simulation framework for evaluating customer service agents across various domains.

Key Features:

• Comprehensive evaluation across multiple task types
• Standardized benchmarking methodology
• Support for diverse model architectures

Quick Start Guide

List the available evaluations:

$ nemo-evaluator ls
tau2_bench:
  * tau2_bench_airline
  * tau2_bench_retail
  * tau2_bench_telecom

NVIDIA NeMo Evaluator provides you with evaluation clients that are specifically built to evaluate model endpoints using our Standard API.

Launching an Evaluation

Run the Evaluation of Your Choice

export API_KEY=your_nvidia_api_key_here

nemo-evaluator run_eval \
    --eval_type tau2_bench_telecom \
    --model_id openai/gpt-oss-120b \
    --model_type chat \
    --model_url https://integrate.api.nvidia.com/v1/chat/completions \
    --api_key_name API_KEY \
    --output_dir './results/tau2bench'

3rd Party Source Code

Users can download the third party source code through the URL provided in the container's README located in workdir.