prompt-testing
Use when comparing two prompt variants, defining quality/efficiency/robustness metrics, or deciding whether to adopt a challenger prompt over a baseline.
How do I install this agent skill?
npx skills add https://github.com/fusengine/claude-plugins --skill prompt-testingIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill provides a framework for A/B testing and measuring AI prompt performance through defined metrics and workflows. It is generally safe but presents a minor risk of indirect prompt injection because it is designed to ingest and process external test datasets which could contain instructions meant to hijack the agent's context.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
What does this agent skill do?
The adoption decision is rule-based: adopt B if its accuracy is at least equal with acceptable token cost, consider B as a trade-off if accuracy improves >10% despite <20% token regression, otherwise keep A or iterate. Requires a minimum of 20 test cases with 15-20% edge cases for statistical significance. </objective>
Prompt Testing
Skill for testing, comparing, and measuring prompt performance.
References
- metrics.md - Load when: defining or scoring Quality/Efficiency/Robustness/UX metrics with thresholds and calculation formulas
- methodology.md - Load when: running a full A/B test (hypothesis, dataset sizing, statistical significance, common pitfalls)
- templates.md - Load when: writing a test dataset JSON or an A/B test report
Testing Workflow
1. DEFINE
└── Test objective
└── Metrics to measure
└── Success criteria
2. PREPARE
└── Variants A and B
└── Test dataset
└── Baseline (if existing)
3. EXECUTE
└── Run on dataset
└── Collect results
└── Document observations
4. ANALYZE
└── Calculate metrics
└── Compare variants
└── Identify patterns
5. DECIDE
└── Recommendation
└── Statistical confidence
└── Next iterations
Performance Metrics
Quality
| Metric | Description | Calculation |
|---|---|---|
| Accuracy | Correct responses | Correct / Total |
| Compliance | Format adherence | Compliant / Total |
| Consistency | Response stability | 1 - Variance |
| Relevance | Meeting the need | Average score (1-5) |
Efficiency
| Metric | Description | Calculation |
|---|---|---|
| Tokens Input | Prompt size | Token count |
| Tokens Output | Response size | Token count |
| Latency | Response time | ms |
| Cost | Price per request | Tokens × Price |
Robustness
| Metric | Description | Calculation |
|---|---|---|
| Edge Cases | Edge case handling | Passed / Total |
| Jailbreak Resist | Bypass resistance | Blocked / Attempts |
| Error Recovery | Error recovery | Recovered / Errors |
For full definitions, thresholds, and the UX metrics category, see metrics.md. For the test dataset and report formats, see templates.md.
Commands
# Create a test
/prompt test create --name "Test v1" --dataset tests.json
# Run an A/B test
/prompt test run --a prompt_a.md --b prompt_b.md --dataset tests.json
# View results
/prompt test results --id test_001
# Compare two tests
/prompt test compare --tests test_001,test_002
Decision Criteria
When to adopt variant B?
IF:
- Accuracy B >= Accuracy A
AND (Tokens B <= Tokens A * 1.1 OR accuracy improvement > 5%)
AND no regression on edge cases
THEN:
→ Adopt B
ELSE IF:
- Accuracy improvement > 10%
AND token regression < 20%
THEN:
→ Consider B (acceptable trade-off)
ELSE:
→ Keep A or iterate
Best Practices
- Minimum 20 test cases for significance
- Include edge cases (15-20% of dataset)
- Test multiple runs for consistency
- Document hypotheses before testing
- Version the prompts being tested
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/fusengine/claude-plugins/prompt-testing">View prompt-testing on skillZs</a>