skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
googlechrome/modern-web-guidance-src49 installs

grader-replay

Validates changes to graders, grading fixtures, and CSS/DOM/AST helpers by replaying saved agent solutions from historical eval jobs and triaging before/after diffs. Use when modifying grader.ts, test-fixture.ts, run-grader.ts, or parser helpers used by graders. Don't use for flagging or investigating eval jobs (use nightly-eval-investigation) or validating gd dev prompt generation (dev-prompt-canary).

How do I install this agent skill?

npx skills add https://github.com/googlechrome/modern-web-guidance-src --skill grader-replay
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    This skill provides a developer tool for replaying historical agent evaluation jobs to validate changes in grading logic. It automates the retrieval of evaluation data from Google Cloud Storage and uses local Git integration to identify relevant files for testing. While it processes and executes historical agent patches, the behavior is consistent with standard software development and testing workflows.

  • Socketpass

    No alerts

  • Snykpass

    Risk: LOW · No issues

What does this agent skill do?

Historical Eval Replay for Grader Changes

Golden calibration (gd dev <guide> --test-grader) only tests three locked solution patches and zero-passrate.patch (which is empty on almost every guide, i.e., the untouched starting app). Historical eval runs contain diverse, imperfect agent implementations that expose broken assumptions, parser edge cases, and false positives.

Because replay applies today's graders against yesterday's saved outputs, before/after mismatches are expected — treat them as signals to triage, not an automatic pass/fail gate.

1. Run Calibration First

Run the repo's golden + zero-passrate calibration on each modified guide:

gd dev <guide> --test-grader

Never run gd dev without --test-grader during validation, or write mode may overwrite manual grader.ts edits.

2. Pull Jobs (if Missing or Stale) & Replay

scripts/replay.ts checks results/suites/ and ~/guidance-evals/ and picks the newest nightly-* job per configured agent using suite_config.json.

If local jobs are missing or drifted behind current base apps, pull recent nightlies while skipping trace archives and screenshots:

gcloud storage ls gs://guidance-evals/
gcloud storage rsync gs://guidance-evals/<suite_id> ~/guidance-evals/<suite_id> \
  --recursive --exclude ".*\.zip$|.*\.png$"

Then run scripts/replay.ts:

# Auto-scope to target graders changed against origin/main
node .agents/skills/grader-replay/scripts/replay.ts

# Scope to specific guides or target keys
node .agents/skills/grader-replay/scripts/replay.ts --guides css-layout,menu

# Replay all tasks (required when modifying test-fixture.ts, run-grader.ts, or parser deps)
node .agents/skills/grader-replay/scripts/replay.ts --all --concurrency 6

# Pin specific jobs, cap runs, or emit JSON
node .agents/skills/grader-replay/scripts/replay.ts \
  --suites nightly-2026-11-02_03-04-27-codex_cli,nightly-2026-10-31_23-00-02-antigravity_cli \
  --limit 12 \
  --json

Why never run grade.mjs inside a job folder? Saved grade.mjs files hardcode the original runner's home directory, use an older runPlaywright signature, and overwrite the saved *_results.json baseline you are comparing against.

Why stacking matters on Gen 2 targets: Eval runs apply zero-passrate.patch before invoking the agent, so agent.patch is diffed on top of base-apps/<app> + zero-passrate.patch. runPlaywright stacks both patches in a throwaway workspace and throws PatchApplyError (reported as PATCH_CONFLICT) if either fails to apply, while passing each task's baselineTarget (widely, newly, or limited).

3. Triage Every DIFF, PATCH_CONFLICT, or ERROR

Inspect the saved agent.patch (or index.html for Gen 1 tasks), the modified grader.ts, and expectations.md to classify each mismatch:

  • Real regression (fix required): Saved pass -> fail where the saved output legitimately satisfies guide.md and expectations.md, or a check stopped matching valid syntax after a refactor.
  • Intentional grader fix (expected diff):
    • pass -> fail when tightening a check that previously false-passed on unrelated CSS/markup. Confirm in agent.patch that the saved solution missed the requirement.
    • fail -> pass when fixing an overly strict check that rejected a pattern guide.md allows. Confirm in agent.patch that the saved solution complies with guide.md and expectations.md; never loosen a check speculatively when the rejected pattern is not allowed by the guide (or when no valid solution failed it).
  • Timeout flakiness: Checks involving timedOut (pass -> timedOut or timedOut -> pass) point to worker contention rather than grader changes — re-run at lower --concurrency.
  • Title or structure drift (expected diff): Check titles appear as saved -> <missing> and <missing> -> now because tests were renamed or consolidated on main. Check git log -p -n 3 -- <grader> and compare pass/fail totals.
  • Base-app drift (PATCH_CONFLICT): An older agent.patch fails to apply against current src/harness/base-apps/ or zero-passrate.patch. Not a grader bug — re-run against a newer nightly job from GCS.

If you edit a grader.ts after triaging, re-run gd dev <guide> --test-grader and scripts/replay.ts, then finish with pnpm run preflight.

Related Skills

  • project-evals — rules for writing and reviewing expectations.md, task.md, and grader.ts.
  • dev-prompt-canary — empirical validation for edits to gd dev prompts and grader generation templates.
  • nightly-eval-investigation — triaging low-performing tasks across nightly jobs in GCS.

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/googlechrome/modern-web-guidance-src/grader-replay">View grader-replay on skillZs</a>