agent-evaluation
Use when evaluating agent performance, building test frameworks, measuring quality, or asking about "agent evaluation", "LLM-as-judge", "agent testing", "quality metrics", "evaluation rubrics", "agent benchmarks"
How do I install this agent skill?
npx skills add https://github.com/eyadsibai/ltk --skill agent-evaluationIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill provides educational content, rubrics, and code templates for evaluating AI agent performance and quality metrics. No security risks were identified.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
- Runlayerpass
1 file scanned · No issues
What does this agent skill do?
Agent Evaluation Methods
Agent evaluation requires different approaches than traditional software. Agents are non-deterministic, may take different valid paths, and lack single correct answers.
Key Finding: 95% Performance Drivers
Research on BrowseComp found three factors explain 95% of variance:
| Factor | Variance | Implication |
|---|---|---|
| Token usage | 80% | More tokens = better performance |
| Tool calls | ~10% | More exploration helps |
| Model choice | ~5% | Better models multiply efficiency |
Implications: Model upgrades beat token increases. Multi-agent architectures validate.
Multi-Dimensional Rubric
| Dimension | Excellent | Good | Acceptable | Failed |
|---|---|---|---|---|
| Factual accuracy | All correct | Minor errors | Some errors | Wrong |
| Completeness | All aspects | Most aspects | Key aspects | Missing |
| Citation accuracy | All match | Most match | Some match | Wrong |
| Tool efficiency | Optimal | Good | Adequate | Wasteful |
LLM-as-Judge
evaluation_prompt = """
Task: {task_description}
Agent Output: {agent_output}
Ground Truth: {ground_truth}
Evaluate on:
1. Factual accuracy (0-1)
2. Completeness (0-1)
3. Citation accuracy (0-1)
4. Tool efficiency (0-1)
Provide scores and reasoning.
"""
Test Set Design
test_set = [
{"name": "simple", "complexity": "simple",
"input": "What is capital of France?"},
{"name": "medium", "complexity": "medium",
"input": "Compare Apple and Microsoft revenue"},
{"name": "complex", "complexity": "complex",
"input": "Analyze Q1-Q4 sales trends"},
{"name": "very_complex", "complexity": "very_complex",
"input": "Research AI tech, evaluate impact, recommend strategy"}
]
Evaluation Pipeline
def evaluate_agent(agent, test_set):
results = []
for test in test_set:
output = agent.run(test["input"])
scores = llm_judge(output, test)
results.append({
"test": test["name"],
"scores": scores,
"passed": scores["overall"] >= 0.7
})
return results
Complexity Stratification
| Level | Characteristics |
|---|---|
| Simple | Single tool call |
| Medium | Multiple tool calls |
| Complex | Many calls, ambiguity |
| Very Complex | Extended interaction, deep reasoning |
Context Engineering Evaluation
Test context strategies systematically:
- Run agents with different strategies on same tests
- Compare quality scores, token usage, efficiency
- Identify degradation cliffs at different context sizes
Continuous Evaluation
- Run evaluations on all agent changes
- Track metrics over time
- Set alerts for quality drops
- Sample production interactions
Avoiding Pitfalls
| Pitfall | Solution |
|---|---|
| Path overfitting | Evaluate outcomes, not steps |
| Ignoring edge cases | Include diverse scenarios |
| Single metric | Multi-dimensional rubrics |
| Ignoring context | Test realistic context sizes |
| No human review | Supplement automated eval |
Best Practices
- Use multi-dimensional rubrics
- Evaluate outcomes, not specific paths
- Cover complexity levels
- Test with realistic context sizes
- Run evaluations continuously
- Supplement LLM with human review
- Track metrics for trends
- Set clear pass/fail thresholds
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/eyadsibai/ltk/agent-evaluation">View agent-evaluation on skillZs</a>