nemo-evaluator
Use when evaluating LLMs, running benchmarks like MMLU/HumanEval/GSM8K, setting up evaluation pipelines, or asking about "NeMo Evaluator", "LLM benchmarking", "model evaluation", "MMLU", "HumanEval", "GSM8K", "benchmark harnesses"
How do I install this agent skill?
npx skills add https://github.com/eyadsibai/ltk --skill nemo-evaluatorIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill provides instructions for using the NeMo Evaluator SDK to benchmark large language models across various harnesses. It involves standard procedures like package installation, API configuration, and CLI execution. No malicious activity was detected.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
- Runlayerwarn
1/1 file flagged
What does this agent skill do?
NeMo Evaluator SDK - Enterprise LLM Benchmarking
Quick Start
NeMo Evaluator SDK evaluates LLMs across 100+ benchmarks from 18+ harnesses using containerized, reproducible evaluation with multi-backend execution (local Docker, Slurm HPC, Lepton cloud).
Installation:
pip install nemo-evaluator-launcher
Basic evaluation:
export NGC_API_KEY=nvapi-your-key-here
cat > config.yaml << 'EOF'
defaults:
- execution: local
- deployment: none
- _self_
execution:
output_dir: ./results
target:
api_endpoint:
model_id: meta/llama-3.1-8b-instruct
url: https://integrate.api.nvidia.com/v1/chat/completions
api_key_name: NGC_API_KEY
evaluation:
tasks:
- name: ifeval
EOF
nemo-evaluator-launcher run --config-dir . --config-name config
Common Workflows
Workflow 1: Standard Model Evaluation
Checklist:
- [ ] Configure API endpoint (NVIDIA Build or self-hosted)
- [ ] Select benchmarks (MMLU, GSM8K, IFEval, HumanEval)
- [ ] Run evaluation
- [ ] Check results
Step 1: Configure endpoint
For NVIDIA Build:
target:
api_endpoint:
model_id: meta/llama-3.1-8b-instruct
url: https://integrate.api.nvidia.com/v1/chat/completions
api_key_name: NGC_API_KEY
For self-hosted (vLLM, TRT-LLM):
target:
api_endpoint:
model_id: my-model
url: http://localhost:8000/v1/chat/completions
api_key_name: ""
Step 2: Select benchmarks
evaluation:
tasks:
- name: ifeval # Instruction following
- name: gpqa_diamond # Graduate-level QA
env_vars:
HF_TOKEN: HF_TOKEN
- name: gsm8k_cot_instruct # Math reasoning
- name: humaneval # Code generation
Step 3: Run and check results
nemo-evaluator-launcher run --config-dir . --config-name config
nemo-evaluator-launcher status <invocation_id>
cat results/<invocation_id>/<task>/artifacts/results.yml
Workflow 2: Slurm HPC Evaluation
defaults:
- execution: slurm
- deployment: vllm
- _self_
execution:
hostname: cluster.example.com
account: my_slurm_account
partition: gpu
output_dir: /shared/results
walltime: "04:00:00"
nodes: 1
gpus_per_node: 8
deployment:
checkpoint_path: /shared/models/llama-3.1-8b
tensor_parallel_size: 2
data_parallel_size: 4
Workflow 3: Model Comparison
# Same config, different models
nemo-evaluator-launcher run --config-dir . --config-name base_eval \
-o target.api_endpoint.model_id=meta/llama-3.1-8b-instruct
nemo-evaluator-launcher run --config-dir . --config-name base_eval \
-o target.api_endpoint.model_id=mistralai/mistral-7b-instruct-v0.3
# Export results
nemo-evaluator-launcher export <id> --dest mlflow
nemo-evaluator-launcher export <id> --dest wandb
Supported Harnesses
| Harness | Tasks | Categories |
|---|---|---|
| lm-evaluation-harness | 60+ | MMLU, GSM8K, HellaSwag, ARC |
| simple-evals | 20+ | GPQA, MATH, AIME |
| bigcode-evaluation-harness | 25+ | HumanEval, MBPP, MultiPL-E |
| safety-harness | 3 | Aegis, WildGuard |
| vlmevalkit | 6+ | OCRBench, ChartQA, MMMU |
| bfcl | 6 | Function calling v2/v3 |
CLI Reference
| Command | Description |
|---|---|
run | Execute evaluation with config |
status <id> | Check job status |
ls tasks | List available benchmarks |
ls runs | List all invocations |
export <id> | Export results (mlflow/wandb/local) |
kill <id> | Terminate running job |
When to Use vs Alternatives
Use NeMo Evaluator when:
- Need 100+ benchmarks from 18+ harnesses
- Running on Slurm HPC clusters
- Requiring reproducible containerized evaluation
- Evaluating against OpenAI-compatible APIs
Use alternatives instead:
- lm-evaluation-harness: Simpler local evaluation
- bigcode-evaluation-harness: Code-only benchmarks
- HELM: Broader evaluation (fairness, efficiency)
Common Issues
Container pull fails: Configure NGC credentials
docker login nvcr.io -u '$oauthtoken' -p $NGC_API_KEY
Task requires env var: Add to task config
tasks:
- name: gpqa_diamond
env_vars:
HF_TOKEN: HF_TOKEN
Increase parallelism:
-o +evaluation.nemo_evaluator_config.config.params.parallelism=8
-o +evaluation.nemo_evaluator_config.config.params.limit_samples=100
Requirements
- Python 3.10-3.13
- Docker (for local execution)
- NGC API Key (for NVIDIA Build)
- HF_TOKEN (for some benchmarks)
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/eyadsibai/ltk/nemo-evaluator">View nemo-evaluator on skillZs</a>