NVIDIA NeMo Evaluator-compatible container with CoDeC support.
NVIDIA NeMo Evaluator
The goal of NVIDIA NeMo Evaluator is to advance and refine state-of-the-art methodologies for model evaluation, and deliver them as modular evaluation packages (evaluation containers and pip wheels) that teams can use as standardized building blocks.
Quick start guide
NVIDIA NeMo Evaluator provide you with evaluation clients, that are specifically built to evaluate model endpoints using our Standard API.
Launching an evaluation for an LLM
- List the available evaluations:
$ nemo-evaluator ls Available tasks: codec: * mmlu_test * gpqa_diamond * gsm8k_train * gsm8k_test * ifeval * mmlu_pro_test * openai_humaneval_prompt * openai_humaneval_canonical_solution * humaneval_plus_prompt * humaneval_plus_canonical_solution * frames * hellaswag_test * hellaswag_train * aime_2025 * aime_2024 * livecodebench_v1 * livecodebench_v5 * bfcl_v3 * bbq * reward_bench_v1 * reward_bench_v2 * math_500_problem * math_500_solution * swebench_test * swebench_train * hle * ifbench * scicode * terminalbench * taubench - Run the evaluation of your choice:
nemo-evaluator run_eval \ --eval_type mmlu_pro_test \ --model_id meta/llama-3.1-70b-instruct \ --model_url https://integrate.api.nvidia.com/v1/chat/completions \ --model_type completions \ --api_key_name MY_API_KEY \ --output_dir /workspace/results - Gather the results
cat /workspace/results/results.yml
Command-Line Tool
Each package comes pre-installed with a set of command-line tools, designed to simplify the execution of evaluation tasks. Below are the available commands and their usage for contamination-detection (see also the CoDeC paper):
Commands
1. List Evaluation Types
nemo-evaluator ls
Displays the evaluation types available within the harness.
2. Run an evaluation
The nemo-evaluator run_eval command executes the evaluation process. Below are the flags and their descriptions:
Required flags
--eval_type <string>The type of evaluation to perform--model_id <string>The name or identifier of the model to evaluate.--model_url <url>The API endpoint where the model is accessible.--model_type <string>The type of the model to evaluate, currently either "chat", "completions", or "vlm".--output_dir <directory>The directory to use as the working directory for the evaluation. The results, including the results.yml output file, will be saved here.
Optional flags
--api_key_name <string>The name of the environment variable that stores the Bearer token for the API, if authentication is required.--run_config <path>Specifies the path to a YAML file containing the evaluation definition.
Example
nemo-evaluator run_eval \
--eval_type aime_2025 \
--model_id my_model \
--model_type completions \
--model_url http://localhost:8000 \
--output_dir ./evaluation_results
If the model API requires authentication, set the API key in an environment variable and reference it using the --api_key_name flag:
export MY_API_KEY="your_api_key_here"
nemo-evaluator run_eval \
--eval_type aime_2025 \
--model_id my_model \
--model_type completions \
--model_url http://localhost:8000 \
--api_key_name MY_API_KEY \
--output_dir ./evaluation_results
Configuring evaluations via YAML
Evaluations in NVIDIA NeMo Evaluator are configured using YAML files that define the parameters and settings required for the evaluation process. These configuration files follow a standard API which ensures consistency across evaluations.
Example of a YAML config:
config:
type: aime_2025
params:
parallelism: 50
limit_samples: 20
target:
api_endpoint:
model_id: meta/llama-3.1-8b-instruct
type: chat
url: https://integrate.api.nvidia.com/v1/chat/completions
api_key: NVIDIA_API_KEY
The priority of overrides is as follows:
- command line arguments
- user config (as seen above)
- task defaults (defined per task type)
- framework defaults
--dry_run option allows you to print the final run configuration and command without executing the evaluation.
Example:
nemo-evaluator run_eval \
--eval_type aime_2025 \
--model_id my_model \
--model_type completions \
--model_url http://localhost:8000 \
--output_dir /tmp/results \
--dry_run
Output:
Rendered config:
command: '{% if target.api_endpoint.api_key is not none %}export API_KEY=${{target.api_endpoint.api_key}}
&& {% endif %} codec --model {{target.api_endpoint.model_id}} --eval_name {{config.params.task}}
--contamination_type {{config.params.extra.contamination_type}} --url {{target.api_endpoint.url}}
--temperature {{config.params.temperature}} --top_p {{config.params.top_p}} --n_context_seeds
{{config.params.extra.n_context_seeds}} --out_dir {{config.output_dir}}/results
--cache_dir {{config.output_dir}}/cache --num_threads {{config.params.parallelism}}
--max_retries {{config.params.max_retries}} --timeout {{config.params.request_timeout}}
--min_length {{config.params.extra.min_length}} --max_length {{config.params.extra.max_length}}
{% if config.params.limit_samples is not none %} --n_samples {{config.params.limit_samples}}{%
endif %}'
config:
output_dir: /tmp/results
params:
extra:
contamination_type: in_context
max_length: 2048
min_length: 100
n_context_seeds: 5
limit_samples: 1000
max_new_tokens: null
max_retries: 10
parallelism: 20
request_timeout: 120
task: aime_2025
temperature: 0.0
top_p: 1.0
supported_endpoint_types:
- completions
type: aime_2025
framework_name: codec
pkg_name: codec
target:
api_endpoint:
adapter_config:
discovery:
dirs: []
modules: []
endpoint_type: completions
interceptors:
- config:
cache_dir: /tmp/results/cache
max_saved_requests: 5
max_saved_responses: 5
reuse_cached_responses: true
save_requests: true
save_responses: true
enabled: true
name: caching
- config: {}
enabled: true
name: endpoint
- config:
cache_dir: /tmp/results/response_stats_cache
logging_aggregated_stats_interval: 100
enabled: true
name: response_stats
log_failed_requests: false
mode: server
post_eval_hooks:
- config:
html_report_size: 5
report_types:
- html
- json
enabled: true
name: post_eval_report
api_key: null
api_key_name: NGC_API_KEY
model_id: my_model
stream: null
type: completions
url: http://localhost:8000/v1/completions
Rendered command:
codec --model my_model --eval_name aime_2025 --contamination_type in_context --url http://localhost:3825 --temperature 0.0 --top_p 1.0 --n_context_seeds 5 --out_dir /tmp/results/results --cache_dir /tmp/results/cache --num_threads 20 --max_retries 10 --timeout 120 --min_length 100 --max_length 2048 --n_samples 1000
FAQ
Deploying a model as an endpoint
NVIDIA NeMo Evaluator utilize a client-server communication architecture to interact with the model. As a prerequisite, the model must be deployed as an endpoint with a NIM-compatible API.
Users have the flexibility to deploy their model using their own infrastructure and tooling.
Servers with APIs that conform to the OpenAI/NIM API standard are expected to work seamlessly out of the box.
List the available evaluations:
$ nemo-evaluator ls
codec:
* aime_2024
* aime_2025
* bbq
* bfcl_v3
* frames
* gpqa_diamond
* gsm8k_test
* gsm8k_train
* hellaswag_test
* hellaswag_train
* hle
* ifbench
* ifeval
* livecodebench_v1
* livecodebench_v5
* math_500_problem
* math_500_solution
* mmlu_pro_test
* mmlu_test
* openai_humaneval
* reward_bench_v1
* reward_bench_v2
* scicode
* swebench_test
* swebench_train
* taubench
* terminalbench
3rd Party Source Code
Users can download the third party source code through the URL provided in the container's README located in workdir.