grader-replay
Validates changes to graders, grading fixtures, and CSS/DOM/AST helpers by replaying saved agent solutions from historical eval jobs and triaging before/after diffs. Use when modifying grader.ts, test-fixture.ts, run-grader.ts, or parser helpers used by graders. Don't use for flagging or investigating eval jobs (use nightly-eval-investigation) or validating gd dev prompt generation (dev-prompt-canary).
How do I install this agent skill?
npx skills add https://github.com/googlechrome/modern-web-guidance-src --skill grader-replayIs this agent skill safe to install?
- Gen Agent Trust Hubpass
This skill provides a developer tool for replaying historical agent evaluation jobs to validate changes in grading logic. It automates the retrieval of evaluation data from Google Cloud Storage and uses local Git integration to identify relevant files for testing. While it processes and executes historical agent patches, the behavior is consistent with standard software development and testing workflows.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
What does this agent skill do?
Historical Eval Replay for Grader Changes
Golden calibration (gd dev <guide> --test-grader) only tests three locked solution patches and zero-passrate.patch (which is empty on almost every guide, i.e., the untouched starting app). Historical eval runs contain diverse, imperfect agent implementations that expose broken assumptions, parser edge cases, and false positives.
Because replay applies today's graders against yesterday's saved outputs, before/after mismatches are expected — treat them as signals to triage, not an automatic pass/fail gate.
1. Run Calibration First
Run the repo's golden + zero-passrate calibration on each modified guide:
gd dev <guide> --test-grader
Never run gd dev without --test-grader during validation, or write mode may overwrite manual grader.ts edits.
2. Pull Jobs (if Missing or Stale) & Replay
scripts/replay.ts checks results/suites/ and ~/guidance-evals/ and picks the newest nightly-* job per configured agent using suite_config.json.
If local jobs are missing or drifted behind current base apps, pull recent nightlies while skipping trace archives and screenshots:
gcloud storage ls gs://guidance-evals/
gcloud storage rsync gs://guidance-evals/<suite_id> ~/guidance-evals/<suite_id> \
--recursive --exclude ".*\.zip$|.*\.png$"
Then run scripts/replay.ts:
# Auto-scope to target graders changed against origin/main
node .agents/skills/grader-replay/scripts/replay.ts
# Scope to specific guides or target keys
node .agents/skills/grader-replay/scripts/replay.ts --guides css-layout,menu
# Replay all tasks (required when modifying test-fixture.ts, run-grader.ts, or parser deps)
node .agents/skills/grader-replay/scripts/replay.ts --all --concurrency 6
# Pin specific jobs, cap runs, or emit JSON
node .agents/skills/grader-replay/scripts/replay.ts \
--suites nightly-2026-11-02_03-04-27-codex_cli,nightly-2026-10-31_23-00-02-antigravity_cli \
--limit 12 \
--json
Why never run grade.mjs inside a job folder? Saved grade.mjs files hardcode the original runner's home directory, use an older runPlaywright signature, and overwrite the saved *_results.json baseline you are comparing against.
Why stacking matters on Gen 2 targets: Eval runs apply zero-passrate.patch before invoking the agent, so agent.patch is diffed on top of base-apps/<app> + zero-passrate.patch. runPlaywright stacks both patches in a throwaway workspace and throws PatchApplyError (reported as PATCH_CONFLICT) if either fails to apply, while passing each task's baselineTarget (widely, newly, or limited).
3. Triage Every DIFF, PATCH_CONFLICT, or ERROR
Inspect the saved agent.patch (or index.html for Gen 1 tasks), the modified grader.ts, and expectations.md to classify each mismatch:
- Real regression (fix required): Saved
pass -> failwhere the saved output legitimately satisfiesguide.mdandexpectations.md, or a check stopped matching valid syntax after a refactor. - Intentional grader fix (expected diff):
pass -> failwhen tightening a check that previously false-passed on unrelated CSS/markup. Confirm inagent.patchthat the saved solution missed the requirement.fail -> passwhen fixing an overly strict check that rejected a patternguide.mdallows. Confirm inagent.patchthat the saved solution complies withguide.mdandexpectations.md; never loosen a check speculatively when the rejected pattern is not allowed by the guide (or when no valid solution failed it).
- Timeout flakiness: Checks involving
timedOut(pass -> timedOutortimedOut -> pass) point to worker contention rather than grader changes — re-run at lower--concurrency. - Title or structure drift (expected diff): Check titles appear as
saved -> <missing>and<missing> -> nowbecause tests were renamed or consolidated onmain. Checkgit log -p -n 3 -- <grader>and compare pass/fail totals. - Base-app drift (
PATCH_CONFLICT): An olderagent.patchfails to apply against currentsrc/harness/base-apps/orzero-passrate.patch. Not a grader bug — re-run against a newer nightly job from GCS.
If you edit a grader.ts after triaging, re-run gd dev <guide> --test-grader and scripts/replay.ts, then finish with pnpm run preflight.
Related Skills
project-evals— rules for writing and reviewingexpectations.md,task.md, andgrader.ts.dev-prompt-canary— empirical validation for edits togd devprompts and grader generation templates.nightly-eval-investigation— triaging low-performing tasks across nightly jobs in GCS.
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/googlechrome/modern-web-guidance-src/grader-replay">View grader-replay on skillZs</a>