skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
fusengine/claude-plugins55 installs

prompt-testing

Use when comparing two prompt variants, defining quality/efficiency/robustness metrics, or deciding whether to adopt a challenger prompt over a baseline.

How do I install this agent skill?

npx skills add https://github.com/fusengine/claude-plugins --skill prompt-testing
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    The skill provides a framework for A/B testing and measuring AI prompt performance through defined metrics and workflows. It is generally safe but presents a minor risk of indirect prompt injection because it is designed to ingest and process external test datasets which could contain instructions meant to hijack the agent's context.

  • Socketpass

    No alerts

  • Snykpass

    Risk: LOW · No issues

What does this agent skill do?

<objective> Prompt Testing runs A/B comparisons between prompt variants through a 5-step workflow: define the objective and metrics, prepare variants A/B and a test dataset, execute on the dataset, analyze and compare results, then decide. Metrics span three categories -- quality (accuracy, compliance, consistency, relevance), efficiency (input/output tokens, latency, cost), and robustness (edge-case handling, jailbreak resistance, error recovery) -- plus a UX category detailed in `metrics.md`.

The adoption decision is rule-based: adopt B if its accuracy is at least equal with acceptable token cost, consider B as a trade-off if accuracy improves >10% despite <20% token regression, otherwise keep A or iterate. Requires a minimum of 20 test cases with 15-20% edge cases for statistical significance. </objective>

Prompt Testing

Skill for testing, comparing, and measuring prompt performance.

References

  • metrics.md - Load when: defining or scoring Quality/Efficiency/Robustness/UX metrics with thresholds and calculation formulas
  • methodology.md - Load when: running a full A/B test (hypothesis, dataset sizing, statistical significance, common pitfalls)
  • templates.md - Load when: writing a test dataset JSON or an A/B test report

Testing Workflow

1. DEFINE
   └── Test objective
   └── Metrics to measure
   └── Success criteria

2. PREPARE
   └── Variants A and B
   └── Test dataset
   └── Baseline (if existing)

3. EXECUTE
   └── Run on dataset
   └── Collect results
   └── Document observations

4. ANALYZE
   └── Calculate metrics
   └── Compare variants
   └── Identify patterns

5. DECIDE
   └── Recommendation
   └── Statistical confidence
   └── Next iterations

Performance Metrics

Quality

MetricDescriptionCalculation
AccuracyCorrect responsesCorrect / Total
ComplianceFormat adherenceCompliant / Total
ConsistencyResponse stability1 - Variance
RelevanceMeeting the needAverage score (1-5)

Efficiency

MetricDescriptionCalculation
Tokens InputPrompt sizeToken count
Tokens OutputResponse sizeToken count
LatencyResponse timems
CostPrice per requestTokens × Price

Robustness

MetricDescriptionCalculation
Edge CasesEdge case handlingPassed / Total
Jailbreak ResistBypass resistanceBlocked / Attempts
Error RecoveryError recoveryRecovered / Errors

For full definitions, thresholds, and the UX metrics category, see metrics.md. For the test dataset and report formats, see templates.md.

Commands

# Create a test
/prompt test create --name "Test v1" --dataset tests.json

# Run an A/B test
/prompt test run --a prompt_a.md --b prompt_b.md --dataset tests.json

# View results
/prompt test results --id test_001

# Compare two tests
/prompt test compare --tests test_001,test_002

Decision Criteria

When to adopt variant B?

IF:
  - Accuracy B >= Accuracy A
  AND (Tokens B <= Tokens A * 1.1 OR accuracy improvement > 5%)
  AND no regression on edge cases
THEN:
  → Adopt B

ELSE IF:
  - Accuracy improvement > 10%
  AND token regression < 20%
THEN:
  → Consider B (acceptable trade-off)

ELSE:
  → Keep A or iterate

Best Practices

  1. Minimum 20 test cases for significance
  2. Include edge cases (15-20% of dataset)
  3. Test multiple runs for consistency
  4. Document hypotheses before testing
  5. Version the prompts being tested

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/fusengine/claude-plugins/prompt-testing">View prompt-testing on skillZs</a>