NVIDIA NeMo Evaluator

The goal of NVIDIA NeMo Evaluator is to advance and refine state-of-the-art methodologies for model evaluation, and deliver them as modular evaluation packages (evaluation containers and pip wheels) that teams can use as standardized building blocks.

agentic_eval evaluates agentic and RAG interactions using Ragas (v0.2.14) metrics:

• topic_adherence — ability of the agent to stay on predefined domains during interactions
• tool_call_accuracy — performance in identifying and calling the required tools (does not use an LLM judge)
• agent_goal_accuracy — ability to identify and achieve user goals (with_reference and without_reference variants)
• answer_accuracy — agreement between a model's response and a reference ground truth
• trajectory_evaluation — quality of the agent's decision-making across its entire trajectory of actions

Quick start guide

NVIDIA NeMo Evaluator provide you with evaluation clients, that are specifically built to evaluate model endpoints using our Standard API.

Launching an evaluation for an LLM

List the available evaluations:

$ nemo-evaluator ls
agentic_eval:
  * agentic_eval_answer_accuracy
  * agentic_eval_goal_accuracy_with_r...
  * agentic_eval_goal_accuracy_withou...
  * agentic_eval_tool_call_accuracy
  * agentic_eval_topic_adherence
  * agentic_eval_trajectory_evaluation

Run the evaluation of your choice:

nemo-evaluator run_eval \
    --eval_type agentic_eval_tool_call_accuracy \
    --model_id meta/llama-3.1-70b-instruct \
    --model_url https://integrate.api.nvidia.com/v1/chat/completions \
    --model_type chat \
    --api_key_name MY_API_KEY \
    --output_dir /workspace/results

Gather the results:

cat /workspace/results/results.yml

Command-Line Tool

Each package comes pre-installed with a set of command-line tools, designed to simplify the execution of evaluation tasks. Below are the available commands and their usage for the agentic_eval:

Commands

1. List Evaluation Types

nemo-evaluator ls

Displays the evaluation types available within the harness.

2. Run an evaluation

The nemo-evaluator run_eval command executes the evaluation process. Below are the flags and their descriptions:

Required flags

• --eval_type <string> The type of evaluation to perform
• --model_id <string> The name or identifier of the model to evaluate.
• --model_url <url> The API endpoint where the model is accessible.
• --model_type <string> The type of the model to evaluate, currently either "chat", "completions", or "vlm".
• --output_dir <directory> The directory to use as the working directory for the evaluation. The results, including the results.yml output file, will be saved here.

Optional flags

• --api_key_name <string> The name of the environment variable that stores the Bearer token for the API, if authentication is required.
• --run_config <path> Specifies the path to a YAML file containing the evaluation definition.

Example

nemo-evaluator run_eval \
    --eval_type agentic_eval_tool_call_accuracy \
    --model_id my_model \
    --model_type chat \
    --model_url http://localhost:8000 \
    --output_dir ./evaluation_results

If the model API requires authentication, set the API key in an environment variable and reference it using the --api_key_name flag:

export MY_API_KEY="your_api_key_here"

nemo-evaluator run_eval \
    --eval_type agentic_eval_tool_call_accuracy \
    --model_id my_model \
    --model_type chat \
    --model_url http://localhost:8000 \
    --api_key_name MY_API_KEY \
    --output_dir ./evaluation_results

Configuring evaluations via YAML

Evaluations in NVIDIA NeMo Evaluator are configured using YAML files that define the parameters and settings required for the evaluation process. These configuration files follow a standard API which ensures consistency across evaluations.

Example of a YAML config:

config:
  type: agentic_eval_tool_call_accuracy  # or other metric types
  params:
    parallelism: 10
    extra:
      judge_model_type: openai
      judge_model_args:
        temperature: 0.1
        max_tokens: 1024
        max_retries: 10
        request_timeout: 300
      judge_sanity_check: True
      dataset_path: "/path/to/dataset.jsonl"
      data_template_path: "/path/to/template.json"  # optional
target:
  api_endpoint:
    api_key: "OPENAI_API_KEY"

The priority of overrides is as follows:

1. command line arguments
2. user config (as seen above)
3. task defaults (defined per task type)
4. framework defaults

--dry_run option allows you to print the final run configuration and command without executing the evaluation.

Configuration parameters

• config:
  • type: The metric type to evaluate (e.g., agentic_eval_tool_call_accuracy)
  • params
    • parallelism: Number of parallel evaluation tasks (default: 10)
    • extra
      • judge_model_type: The type of judge model to use (either openai or nvidia-nim; default: openai)
      • judge_model_args: Judge model configuration (all optional): temperature, max_tokens, max_retries, request_timeout, reasoning_effort, and any other parameters supported by the ChatOpenAI client
      • judge_sanity_check: Enable/disable judge model sanity checks (default: True)
      • judge_raise_exceptions: Raise exceptions when the judge model fails, otherwise return NaN score (default: False)
      • dataset_path: Path to the dataset file
      • metric_mode: Specific mode for topic_adherence metric (precision, recall, or f1; default: f1)
      • data_template_path: Path to a data template file for custom data formats (optional)
      • trajectory_used_tools: Comma-separated list of tool names available to the agent (required for trajectory_evaluation)
      • trajectory_custom_tools: JSON object mapping custom tool names to their descriptions (optional for trajectory_evaluation)
• target: As a work-around for reasoning judge models, part of the judge model configuration is passed in the target section.
  • api_endpoint:
    • api_key: environment variable that holds the API key for the judge model (optional for nvidia-nim)
    • model_id: The LLM model to use as a judge (optional for openai; default: gpt-4o)
    • url: Base URL for the judge model API (optional for openai)

FAQ

Deploying a model as an endpoint

NVIDIA NeMo Evaluator utilize a client-server communication architecture to interact with the model. As a prerequisite, the model must be deployed as an endpoint with a NIM-compatible API.

Users have the flexibility to deploy their model using their own infrastructure and tooling.

Servers with APIs that conform to the OpenAI/NIM API standard are expected to work seamlessly out of the box.

Preparing your dataset

agentic_eval evaluates custom data loaded from a JSONL file, with each entry representing a user interaction that conforms to the Ragas format. User input is a list of conversation messages of three types — human, ai, and tool — where each message has content, type, and (for ai messages) optional tool_calls.

Example entry for tool_call_accuracy:

{
    "user_input": [
        {"content": "", "type": "human"},
        {"content": "", "type": "ai", "tool_calls": [{"name": "", "args": {"": ""}}]},
        {"content": "", "type": "tool"},
        {"content": "", "type": "ai"}
    ],
    "reference_tool_calls": [
        {"name": "", "args": {"": ""}}
    ]
}

Other metrics use slightly different fields — topic_adherence expects reference_topics, agent_goal_accuracy_with_reference expects reference, and answer_accuracy expects user_input/response/reference. trajectory_evaluation uses the NAT (NVIDIA Agent Toolkit) result.json format with intermediate_steps.

If your data uses different keys, provide a data_template_path pointing to a JSON template that maps your columns to the expected schema, e.g.:

{
    "user_input": "{{ item.messages | tojson }}",
    "reference_tool_calls": "{{ item.reference | tojson }}"
}

Configuring the LLM judge

topic_adherence, agent_goal_accuracy_with_reference, agent_goal_accuracy_without_reference, answer_accuracy, and trajectory_evaluation use an LLM judge. Two judge types are supported: openai and nvidia-nim. If no judge model is provided, OpenAI gpt-4o is used by default and an API key is required.

For the answer_accuracy metric, Ragas recommends max_tokens: 10 (the judge outputs single-token scores) and temperature: 0.1 for consistent scoring.

To use an OpenAI reasoning model as the judge, set model_id to e.g. o3-mini and optionally set config.params.extra.judge_model_args.reasoning_effort (low, medium, or high). To use an NVIDIA Nemotron reasoning model, configure the target section:

target:
  api_endpoint:
    adapter_config:
      use_reasoning: True
      end_reasoning_token: "</think>"
      use_system_prompt: True
      custom_system_prompt: "detailed thinking on"

License

This container is released under the Apache License, Version 2.0. See https://www.apache.org/licenses/LICENSE-2.0 for the full text.

3rd Party Source Code

Users can download the third party source code through the URL provided in the container's README located in workdir.