test-evidence-review

!bash "${CLAUDE_SKILL_DIR}/../../hooks/yaml-helper.sh" resolve_config --keys automation,workflow,qa.level,testing.strict

Automation mode: Resolve modes.automation (project.local.yaml → project.yaml → default collaborative). Every AskUserQuestion call and every file write follows .claude/docs/automation-modes.md (collaborative asks always · guided major-only · autonomous logs and proceeds; automation_always_ask categories always prompt).

qa.level and testing.strict (resolved above) decide what this review may call a blocker — the same rules /story-done closes stories by, so a BLOCKING item here is one that would stop /story-done:

• qa.level: minimal waives tests. A Logic, Integration or Config/Data story with no test evidence (for Config/Data, no smoke report) is WAIVED (qa.level: minimal) — not MISSING — and raises no BLOCKING item; a test that does exist is still reviewed, but its gaps are ADVISORY. Visual/Feel and UI evidence (the retained screenshots, and the Visual/Feel sign-off) is never waived.
• Gate level per story type: take testing.strict.<type> from the resolved block (Logic→logic, Integration→integration, Visual/Feel→visual, UI→ui, Config/Data→config): true → BLOCKING, false → ADVISORY, unset → a plain-boolean testing.strict: true|false (legacy form) applies to every type; else the Default Gate Level in .claude/docs/coding-standards.md (BLOCKING for Logic, Integration, Visual/Feel and UI; ADVISORY for Config/Data). Every "BLOCKING" below means that story type's gate level.

Test Evidence Review

/smoke-check verifies that test files exist and pass. This skill goes further — it reviews the quality of those tests and evidence documents. A test file that exists and passes may still leave critical behaviour uncovered. A manual evidence doc that exists may lack the sign-offs required for closure.

Output: Summary report (in conversation) + optional production/qa/evidence-review-[date].md

When to run:

• Before QA hand-off sign-off (/team-qa Phase 5)
• On any story where test quality is in question
• As part of milestone review for Logic and Integration story quality audit

---

1. Parse Arguments

Modes:

• /test-evidence-review [story-path] — review a single story's evidence
• /test-evidence-review sprint — review all stories in the current sprint
• /test-evidence-review [system-name] — review all stories in an epic/system
• No argument — ask which scope: "Single story", "Current sprint", "A system"

---

2. Load Stories in Scope

Based on the argument:

Single story: Read the story file directly. Extract: Story Type, Test Evidence section, story slug, system name.

Sprint: Read the most recently modified file in production/sprints/; extract the list of story file paths from the sprint plan.

System: Glob production/epics/[system-name]/story-*.md.

If the resolved scope contains ZERO stories, stop here. Report NOT ASSESSED — no stories in scope, name which scope was searched and which path was empty, and route: no sprint file → /sprint-plan new (at workflow: minimal, which has no sprints, /test-evidence-review [epic-slug] instead); a sprint plan listing no stories → /create-stories [epic-slug]; a [system-name] glob that matched nothing → name the glob. Do not continue to Section 3.

Guard the empty scope, not just the per-story unknown. The verdict vocabulary here — ADEQUATE / INCOMPLETE / MISSING — needs a "could not check" value, or an unverifiable story acquires a verdict claiming somebody verified it; that is what NOT ASSESSED is for. But giving the per-story unknown a home does nothing for the empty-scope unknown. With no stories the Section 6 report renders an empty Summary table and ends BLOCKING items: 0 / ADVISORY items: 0 — which reads as everything reviewed, all fine. This skill gates story closure, and coding-standards.md marks Logic, Integration, Visual/Feel and UI evidence BLOCKING, so a false-clean closes stories nobody reviewed.

This is a recurring shape: the sophisticated inner rule present, the outer boundary unguarded. Ask it of any skill that aggregates — what does this emit when the set is empty?

For the resulting story set, collect the fields below with targeted section greps, not a full read of each story:

Grep pattern="## Test Evidence" glob="production/epics/**/story-*.md" output_mode="content" -A 8
Grep pattern="## Acceptance Criteria" glob="production/epics/**/story-*.md" output_mode="content" -A 15

• Story Type (Logic / Integration / Visual/Feel / UI / Config/Data) and the stated evidence path — both live under ## Test Evidence, so the first grep's -A 8 captures them.
• Acceptance Criteria list — the ## Acceptance Criteria block from the second grep.
• Story slug (from the file name) and System (from the directory path) — no read. Full-read a story only when its Test Evidence section is missing or ambiguous. (In Sprint mode, scope the globs to the sprint plan's story paths.)

---

3. Locate Evidence Files

For each story, find the evidence. Check the story's stated evidence path first — the one Section 2 captured from its ## Test Evidence section. If it exists, use it. Fall back to searching only when no path is stated or it is not found:

Logic stories: search the engine's unit-test root — tests/unit/[system]/ (Godot), Assets/Tests/EditMode/ (Unity), Source/<Module>/Private/Tests/ (Unreal); see .claude/docs/directory-structure.md — for a file whose name contains the story slug or its system. Test files are named per .claude/rules/test-standards.md, not necessarily [story-slug]_test.*.

Integration stories: the same, under the integration root (tests/integration/[system]/, Assets/Tests/PlayMode/, or Unreal's test root)

• Also check production/session-logs/ for playtest records mentioning the story

Visual/Feel and UI stories: Glob production/qa/evidence/ for a retained image (*.png, *.jpg, *.gif) belonging to this story, and for [story-slug]-evidence.*

Config/Data stories: Glob production/qa/smoke-*.md (any smoke check report)

Note what was found (path) or not found (gap) for each story.

---

4. Review Automated Test Quality (Logic / Integration)

For each test file found, read it and evaluate:

Assertion coverage

Count the number of distinct assertions (lines containing assert, expect, check, verify, or engine-specific assertion patterns). Low assertion count is a quality signal — a test that makes only 1 assertion per test function may not cover the range of expected behaviour.

Thresholds:

• 3+ assertions per test function → normal
• 1-2 assertions per test function → note as potentially thin
• 0 assertions (test exists but no asserts) → flag as BLOCKING (the story type's gate level; ADVISORY at qa.level: minimal) — the test passes vacuously and proves nothing

Edge case coverage

For each acceptance criterion in the story that contains a number, threshold, or "when X happens" conditional: check whether a test function name or test body references that specific case.

Heuristics:

• Grep test file for "zero", "max", "null", "empty", "min", "invalid", "boundary", "edge" — presence of any is a positive signal
• If the story has a Formulas section with specific bounds: check whether tests exercise at minimum/maximum values

Naming quality

Test function names should describe: the scenario + the expected result. Pattern per engine (.claude/rules/test-standards.md): Godot test_[scenario]_[expected], Unity [Scenario]_[Expected], Unreal <Project>.[System].[Scenario].

Flag functions named generically (test_1, test_run, testBasic) as naming issues — they make failures harder to diagnose.

Formula traceability

For Logic stories where the GDD has a Formulas section: check that the test file contains at least one test whose name or comment references the formula name or a formula value. A test that exercises a formula without mentioning it by name is harder to maintain when the formula changes.

---

5. Review Manual Evidence Quality (Visual/Feel / UI)

For each evidence document found, read it and evaluate the checks below.

A UI story needs no evidence document. When it has none, its retained screenshots are the evidence: apply only Screenshot / artefact completeness (an image of each screen the story changes — a screen a criterion only navigates to, such as the title screen after Quit, needs none of its own) and Date coverage (the images' dates). A UI story is ADEQUATE on those two — never INCOMPLETE for lacking a doc, its criterion linkage or a sign-off.

Criterion linkage

The evidence doc should reference each acceptance criterion from the story. Check: does the evidence doc contain each criterion (or a clear rephrasing)? Missing criteria mean a criterion was never verified.

Sign-off completeness

What closes the story follows the evidence table in .claude/docs/coding-standards.md — the same rule /story-done applies, at every qa.level:

• Visual/Feel: the evidence doc (from the test-evidence template) must have every sign-off row [x] Approved. A blank or missing row is INCOMPLETE.
• UI: no sign-off is required — the retained screenshots close it. Note a sign-off if one exists; never mark a UI story INCOMPLETE for lacking one.

Screenshot / artefact completeness

For Visual/Feel stories: Glob production/qa/evidence/ for a retained image (*.png, *.jpg, *.gif) belonging to this story. A doc that describes a visual check but retains no image is INCOMPLETE — this gate is BLOCKING by default, and a description is an assertion rather than evidence.

For UI stories: require the same retained screenshot of each screen touched. A walkthrough log is welcome but not required.

Date coverage

Evidence doc should have a date. If the date is earlier than the story's last major change (heuristic: compare against sprint start date from the sprint plan), flag as POTENTIALLY STALE — the evidence may not cover the final implementation.

---

6. Build the Review Report

For each story, assign a verdict:

Verdict	Meaning
ADEQUATE	Test/evidence exists, passes quality checks, all criteria covered
INCOMPLETE	Test/evidence exists but has quality gaps (thin assertions, missing sign-offs)
MISSING	No test or evidence found for a story type that requires it at this qa.level
NOT ASSESSED	The review could not be performed for this story — see below
WAIVED (qa.level: minimal)	A Logic, Integration or Config/Data story with no test evidence at qa.level: minimal — tests are not required, so nothing is missing; name it as waived

NOT ASSESSED is required, and it is not a softer MISSING. The other three verdicts all presume the review actually ran. MISSING means I looked and there was nothing there — a real, reportable failure. It must never be used for I could not look, which is not a finding about the story at all. Use NOT ASSESSED when:

• the story's type cannot be determined, so the required evidence is unknown (the type→evidence mapping in .claude/docs/coding-standards.md is what makes any other verdict meaningful);
• the evidence path is named but unreadable or outside this run's scope;
• the story file itself could not be parsed.

Say which of those it was, per story. A reader cannot act on a bare NOT ASSESSED, and the whole point of separating it from MISSING is that the two have different fixes: one needs a test written, the other needs the reviewer given access or the story classified.

The overall sprint/system verdict is the worst story verdict present, and NOT ASSESSED outranks ADEQUATE: a run that could not assess part of its scope has not established that the scope is adequate. It does not outrank INCOMPLETE or MISSING — a known failure is more actionable than an unknown, and demoting a real failure behind an access problem would bury it. WAIVED ranks below ADEQUATE: the overall verdict reads WAIVED only when every story in scope was waived, and the report names each waived story either way.

Why this needed saying: evidence review gates story completion, and coding-standards.md makes Logic, Integration, Visual/Feel and UI evidence BLOCKING. A verdict vocabulary with no way to express "I could not check" forces every unknown into one of three claims about the story — which is how a story nobody verified acquires a verdict that says somebody did.

## Test Evidence Review

> **Date**: [date]
> **Scope**: [single story path | Sprint [N] | [system name]]
> **Stories reviewed**: [N]
> **Overall verdict**: ADEQUATE / INCOMPLETE / MISSING / NOT ASSESSED / WAIVED
> **qa.level**: [minimal | standard | full] — **testing.strict**: [the resolved per-type values, or "unset — coding-standards.md defaults"]

---

### Story-by-Story Results

#### [Story Title] — [Type] — [ADEQUATE/INCOMPLETE/MISSING/NOT ASSESSED/WAIVED]

**Test/evidence path**: `[path]` (found) / (not found)

**Automated test quality** *(Logic/Integration only)*:
- Assertion coverage: [N per function on average] — [adequate / thin / none]
- Edge cases: [covered / partial / not found]
- Naming: [consistent / [N] generic names flagged]
- Formula traceability: [yes / no — formula names not referenced in tests]

**Manual evidence quality** *(Visual/Feel/UI only)*:
- Criterion linkage: [N/M criteria referenced]
- Lead sign-off: [✓ approved | ✗ missing | deferred — reason] *(Visual/Feel only; a UI story needs none)*
- Artefacts: [screenshots present / missing / N/A]
- Freshness: [dated [date] — current / potentially stale]

**Issues**:
- BLOCKING: [description] *(prevents story-done)*
- ADVISORY: [description] *(should fix before release)*

---

### Summary

| Story | Type | Verdict | Issues |
|-------|------|---------|--------|
| [title] | Logic | ADEQUATE | None |
| [title] | Integration | INCOMPLETE | Thin assertions (avg 1.2/function) |
| [title] | Visual/Feel | INCOMPLETE | Lead sign-off missing |
| [title] | Logic | MISSING | No test file found |
| [title] | Logic | WAIVED | No test — waived at `qa.level: minimal` |

**BLOCKING items** (must resolve before story can be closed): [N]
**ADVISORY items** (should address before release): [N]

---

7. Write Output (Optional)

Present the report in conversation.

Ask: "May I write this test evidence review to production/qa/evidence-review-[date].md?"

This is optional — the report is useful standalone. Write only if the user wants a persistent record.

After the report:

• For BLOCKING items: "These must be resolved before /story-done can mark the story Complete. Would you like to address any of them now?"
• For thin assertions: "Consider running /test-helpers [system] to see scaffolded assertion patterns for common cases."
• For missing sign-offs: "Manual sign-off is required from [role]. Share [evidence-path] with them to complete sign-off."

Verdict: COMPLETE — evidence review finished. Use CONCERNS if BLOCKING items were found.

---

Collaborative Protocol

• Report quality issues, do not fix them — this skill reads and evaluates; it does not modify test files or evidence documents
• ADEQUATE means adequate for shipping, not perfect — avoid nitpicking tests that are functioning and comprehensive enough to give confidence
• BLOCKING vs. ADVISORY distinction is important — only flag BLOCKING when the gap leaves a story criterion genuinely unverified
• Ask before writing — the report file is optional; always confirm before writing