ai-agent-testing
Use this skill when you need to test AI agent goals, state, planning, recovery, and safety boundaries; triggers include ai agent testing.
How do I install this agent skill?
npx skills add https://github.com/naodeng/awesome-qa-skills --skill ai-agent-testingIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill is a framework for testing AI agent behavior, planning, and safety boundaries. It consists of instructional prompts and evaluation scripts. No malicious patterns or security risks were identified.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
What does this agent skill do?
AI Agent Testing
When to Use
- Use this skill when you need to systematically verify agent planning, memory, tool use, recovery, safety boundaries, and task completion quality.
- Use it to review an existing plan, result, or evidence set and produce actionable improvements.
- Use it when context is incomplete but a bounded first pass is still valuable.
Output Format Options
- Default to Markdown for review, execution, and incremental refinement.
- When the user requests tables, CSV, JSON, or ticket fields, preserve risk, evidence, priority, and boundary information.
- For machine-consumed output, confirm the schema, enums, and required fields first.
How to Use
- Read and follow
prompts/ai-agent-testing.md, including its input contract, execution rules, minimum coverage, and output order. - Add only context that changes the decision: scope, environment, version, constraints, evidence, and success criteria.
- Audit the input, then separate confirmed facts, working assumptions, and open questions.
- Rank by risk and evidence strength, and produce an artifact that can be executed or reviewed directly.
- If information is missing, deliver a bounded first pass and state which conclusions remain unsupported.
Reference Files
-
When calibrating domain outcomes, valid controls, or evidence gaps, read references/domain-decision-example.md for sanitized input, per-item oracles, and reproducible calculations.
-
Always read
prompts/ai-agent-testing.md; it is the complete execution specification for this skill. -
For evaluation or regression, read
evals/eval.yamland the relevant cases underevals/cases/. -
Load
references/,examples/,scripts/, oroutput-formats.mdonly when those directories exist and the task needs them.
Domain Execution Essentials
- Define input sample ID, task-success oracle, allowed tools/resources, prohibited effects, and stopping conditions. Missing goals remain open.
- Check run/step order, permission, idempotency, isolation, recovery, and budget invariants. Judge outcome correctness and process compliance separately; either failure prevents an overall pass.
- Classify termination as success, goal failure, budget exhaustion, cancellation, safety block, or environment error. Missing terminal events do not prove completion.
- Record model/prompt/tool/data/environment versions, sample selection, and project-supplied repetitions. Report per-sample pass/fail counts and intermittent failures; a single trial establishes no stability claim.
Core Constraints
- evaluate both final results and execution traces
- use repeated trials for stochastic behavior
- never treat one success as reliability evidence
- Never invent system behavior, fields, data, metrics, or root causes absent from the evidence.
- Link important conclusions to evidence; mark unsupported conclusions as hypotheses with a verification method.
- Explain priority using business impact, likelihood, or detectability.
Delivery Checklist
- Covered: task completion, planning quality, multi-step consistency, tool use, memory contamination, recovery, authorization boundaries, cost and latency.
- Separated facts, assumptions, gaps, and recommendations.
- Gave high-risk items a priority, evidence basis, owner or next action.
- Defined verifiable decision criteria instead of generic advice.
- Performed no unauthorized production writes or destructive actions.
Common Pitfalls
- Listing checks without preconditions, expected outcomes, or evidence.
- Marking everything high priority and avoiding tradeoffs.
- Substituting tool names or generic theory for domain reasoning.
- Refusing incomplete input, or pretending incomplete evidence supports certainty.
Best Practices
- Start with paths most likely to cause business loss, safety issues, or release blockage.
- Reduce uncertainty through the smallest verifiable experiment and record reproduction conditions.
- Make the artifact executable and independently reviewable by another engineer.
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/naodeng/awesome-qa-skills/ai-agent-testing">View ai-agent-testing on skillZs</a>