verify-docs-findable

verify-docs-findable — cold-read the docs with fresh eyes

The insight this operationalizes: complete ≠ findable. An index that lists every doc (and even passes a "every doc is indexed" check) can still make a reader take 5 hops and fall back to source code to answer a basic question. The only honest test is to give the question to someone who has never seen the repo and watch how far they get from the docs alone. A fresh cheap agent is that someone.

Why HAIKU, and why "docs only"

• Haiku (a small, cheap model), fresh (no session context). The point is a naive cold reader, NOT a clever one. A strong model with your context will grep the source, reason around gaps, and hide the doc problem. Haiku, told to use only the docs, exposes it.
• "Use ONLY the docs, start at the index." The moment a verifier has to open source code to answer, the docs have failed — that's the signal. Steps-from-index is the metric.

Procedure

1. Get the questions (3–8)

Test the recurring, load-bearing, and mis-stated questions — the ones that actually cost time. Sources, in order:

• What the user (or you) just got confused about this session — the highest-signal source.
• The "canonical answers" the docs claim to provide (a docs corpus usually has ~5–8 facts it exists to convey; test those).
• If none given: ask the user for the questions, or infer them from the corpus's top-level topics. Don't invent trivia — test what a real contributor needs.

Phrase each as a real question a reader would ask, not a doc title.

2. Spawn one fresh Haiku verifier per question (in parallel)

Use the Agent tool with model: "haiku", run_in_background: true, one per question. Give each the SAME strict template (fill in <QUESTION>):

You are a fresh engineer opening <repo path> for the first time. Answer using ONLY the repo's docs — START at the index (<the index file(s)>) and follow pointers; prefer docs over reading source code. QUESTION: <QUESTION> REPORT: (a) your answer; (b) the exact file(s) where you found it; (c) how many steps from the index (did a pointer take you straight there, or did you dig?); (d) clear/unambiguous or scattered/confusing? (e) did you have to read source code because the docs didn't say? Be honest — this tests whether the docs make this findable.

Run all in one message so they go concurrently. (Haiku + parallel = the audit is cheap.)

3. Score each answer

Signal	Good	Gap
Correct?	matches ground truth	wrong/partial → the doc is wrong or missing
Steps from index	1–2	3+ → no signposted pointer
Dead-ends	none	landed on a wrong-but-plausible doc first → title/scoping is misleading
Needed source code?	no	yes → the docs don't actually state it
Contradictions	—	cited two docs that disagree → a cohesion bug (fix immediately)

Verifiers surface contradictions for free — a cold reader citing two docs that say different things is the cheapest contradiction-finder you have.

4. Fix the gaps

• 3+ steps / dead-end → add a question→doc pointer at the index entry point (a "Canonical answers: question → the ONE doc" block). Naming the answer is what turns 5 hops into 1.
• Needed source code → the fact isn't in prose; write it into the one canonical doc.
• Wrong answer → the doc is stale/incorrect; fix it (append-don't-erase if it records a decision).
• Contradiction → reconcile the docs; keep the historical decision, mark it superseded.
• Scattered across N docs → consolidate into ONE doc per question; make the others point to it.

5. Re-verify (measure, don't assume)

Re-run the worst question(s) against the fixed docs. Confirm the number moved (e.g. 5 steps → 2). A fix you didn't re-measure is a guess.

Output

A short findability scorecard (question | correct | steps | needed-source | verdict) + a prioritized fix list. Report the before/after on any question you fixed and re-verified.

Notes

• This is orthogonal to "is every doc indexed?" completeness checks — it measures the layer above: can a stranger reach the answer. Run it after any doc reorganization.
• Keep the question set in the repo (e.g. a docs-findability-questions.md) so the audit is repeatable and the canonical answers stay honest as the corpus grows.
• Scales down (1 question, 1 agent, to spot-check one fix) and up (the full canonical set).