jurgendn/agent-skills69 installs
model-eval-error-analysis
Evaluate a model rigorously and inspect where it fails. Use after training, during comparison to baselines, for LLM/system evaluations, noisy or suspiciously clean results, or whenever benchmark numbers alone are insufficient and the user needs slices, failure categories, and explanations.
How do I install this agent skill?
npx skills add https://github.com/jurgendn/agent-skills --skill model-eval-error-analysisIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill provides a standard workflow for evaluating AI models and analyzing failure modes. It contains no executable code, remote dependencies, or suspicious instructions.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
What does this agent skill do?
Model Eval Error Analysis
Do not stop at one headline metric.
Intake
Collect:
- task, dataset, split, metric, and baseline;
- model outputs or failure examples if available;
- aggregate numbers with seeds/uncertainty;
- deployment or paper claim the evaluation must support;
- known risky slices.
Workflow
- Verify the metric implementation and data split.
- Report the headline result with provenance.
- Slice performance by relevant factors such as:
- difficulty
- length / scale
- class imbalance
- domain shift
- prompt family
- robustness setting
- Inspect failures manually and cluster them.
- Separate:
- random noise
- systematic failure modes
- evaluator bugs
- Compare against the strongest baseline on the same slices.
- End with concrete next actions.
Rules
- Never smooth away inconvenient variation.
- If confidence intervals or multiple seeds matter, say so.
- A higher average may still hide a worse model for important slices.
Output shape
Return:
- Metrics checked
- Slice analysis
- Failure taxonomy
- Suspected causes
- Recommended fixes or follow-up experiments
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/jurgendn/agent-skills/model-eval-error-analysis">View model-eval-error-analysis on skillZs</a>