expect
Diff-aware browser testing: maps the git diff to affected pages, plans and runs targeted checks via agent-browser. Use when testing UI changes before merge. Not for writing tests (use cover), grading a suite (use verify) or unit tests.
How do I install this agent skill?
npx skills add https://github.com/yonatangross/orchestkit --skill expectIs this agent skill safe to install?
- Gen Agent Trust Hubpass
This skill provides a comprehensive browser testing framework using git diffs to target affected UI components. It includes an optional AI-driven judging feature that communicates with an external API and supports session recording via script injection.
- Socketwarn
2 alerts: gptAnomaly
- Snykwarn
Risk: MEDIUM · 2 issues
What does this agent skill do?
Expect — Diff-Aware AI Browser Testing
Analyze git changes, generate targeted test plans, and execute them via AI-driven browser automation.
Note: If
disableSkillShellExecutionis enabled, the agent-browser install check won't run. Verify it's installed:npx agent-browser --version.
expect # Auto-detect changes, test affected pages
expect -m "test the checkout flow" # Specific instruction
expect --flow login # Replay a saved test flow
expect --target branch # Test all changes on current branch vs main
expect -y # Skip plan review, run immediately
Core principle: Only test what changed. Git diff drives scope — no wasted cycles on unaffected pages.
Argument Resolution
ARGS = "[-m <instruction>] [--target unstaged|branch|commit] [--flow <slug>] [-y]"
# Parse from full argument string
import re
raw = "$ARGUMENTS" # Full argument string
INSTRUCTION = None
TARGET = "unstaged" # Default: test unstaged changes
FLOW = None
SKIP_REVIEW = False
# Extract -m "instruction"
m_match = re.search(r'-m\s+["\']([^"\']+)["\']|-m\s+(\S+)', raw)
if m_match:
INSTRUCTION = m_match.group(1) or m_match.group(2)
# Extract --target
t_match = re.search(r'--target\s+(unstaged|branch|commit)', raw)
if t_match:
TARGET = t_match.group(1)
# Extract --flow
f_match = re.search(r'--flow\s+(\S+)', raw)
if f_match:
FLOW = f_match.group(1)
# Extract -y
if '-y' in raw.split():
SKIP_REVIEW = True
STEP 0: MCP Probe + Prerequisite Check
# memory may be absent from the user's .mcp.json; probe so the memory steps below are skipped, not failed:
ToolSearch(query="select:mcp__memory__search_nodes")
# Verify agent-browser is available (Rust-native, no Playwright)
Bash("command -v agent-browser || npx agent-browser --version")
# If missing: "Install agent-browser: npm i -g agent-browser"
# Load agent-browser's version-matched workflow guide (ships with the CLI).
# NOT `skills get agent-browser` — that resolves but returns only the thin
# top-level router; `core --full` is the actual 2,800+ line command reference.
Bash("agent-browser skills get core --full")
CRITICAL: Task Management
# 1. Create main task IMMEDIATELY
TaskCreate(
subject="Expect: test changed code",
description="Diff-aware browser testing pipeline",
activeForm="Running diff-aware browser tests"
)
# 2. Create subtasks for each pipeline phase
TaskCreate(subject="Check fingerprint (skip if unchanged)", activeForm="Checking fingerprint") # id=2
TaskCreate(subject="Scan git diff and classify changes", activeForm="Scanning diff") # id=3
TaskCreate(subject="Map changes to routes/URLs", activeForm="Mapping routes") # id=4
TaskCreate(subject="Generate AI test plan", activeForm="Generating test plan") # id=5
TaskCreate(subject="Execute tests via agent-browser", activeForm="Executing browser tests") # id=6
TaskCreate(subject="Compile test report", activeForm="Compiling report") # id=7
# 3. Set dependencies for sequential phases
TaskUpdate(taskId="3", addBlockedBy=["2"]) # Diff scan needs fingerprint check
TaskUpdate(taskId="4", addBlockedBy=["3"]) # Route map needs diff results
TaskUpdate(taskId="5", addBlockedBy=["4"]) # Test plan needs route map
TaskUpdate(taskId="6", addBlockedBy=["5"]) # Execution needs test plan
TaskUpdate(taskId="7", addBlockedBy=["6"]) # Report needs execution results
# 4. Update status as you progress
TaskUpdate(taskId="2", status="in_progress") # When starting
TaskUpdate(taskId="2", status="completed") # When done — repeat for each subtask
Pipeline Overview
Git Diff → Route Map → Fingerprint Check → Test Plan → Execute → Report
| Phase | What | Output | Reference |
|---|---|---|---|
| 1. Fingerprint | SHA-256 hash of changed files | Skip if unchanged since last run | references/fingerprint.md |
| 2. Diff Scan | Parse git diff, classify changes | ChangesFor data (files, components, routes) | references/diff-scanner.md |
| 3. Route Map | Map changed files to affected pages/URLs | Scoped page list | references/route-map.md |
| 4. Test Plan | Generate AI test plan from diff + route map | Markdown test plan with steps | references/test-plan.md |
| 5. Execute | Run test plan via agent-browser | Pass/fail per step, screenshots | references/execution.md |
| 6. Report | Aggregate results, artifacts, exit code | Structured report + artifacts | references/report.md |
Phase 1: Fingerprint Check
Check if the current changes have already been tested:
Read(".expect/fingerprints.json") # Previous run hashes
# Compare SHA-256 of changed files against stored fingerprints
# If match: "No changes since last test run. Use --force to re-run."
# If no match or --force: continue to Phase 2
Load: Read("references/fingerprint.md")
Phase 2: Diff Scan
Analyze git changes based on --target:
if TARGET == "unstaged":
diff = Bash("git diff")
files = Bash("git diff --name-only")
elif TARGET == "branch":
diff = Bash("git diff main...HEAD")
files = Bash("git diff main...HEAD --name-only")
elif TARGET == "commit":
diff = Bash("git diff HEAD~1")
files = Bash("git diff HEAD~1 --name-only")
Classify each changed file into 3 levels:
- Direct — the file itself changed
- Imported — a file that imports the changed file
- Routed — the page/route that renders the changed component
Load: Read("references/diff-scanner.md")
Phase 3: Route Map
Map changed files to testable URLs using .expect/config.yaml:
# .expect/config.yaml
base_url: http://localhost:3000
route_map:
"src/components/Header.tsx": ["/", "/about", "/pricing"]
"src/app/auth/**": ["/login", "/signup", "/forgot-password"]
"src/app/dashboard/**": ["/dashboard"]
If no route map exists, infer from Next.js App Router / Pages Router conventions.
Load: Read("references/route-map.md")
Phase 4: Test Plan Generation
Build an AI test plan scoped to the diff, using the scope strategy for the current target:
scope_strategy = get_scope_strategy(TARGET) # See references/scope-strategy.md
prompt = f"""
{scope_strategy}
Changes: {diff_summary}
Affected pages: {affected_urls}
Instruction: {INSTRUCTION or "Test that the changes work correctly"}
Generate a test plan with:
1. Page-level checks (loads, no console errors, correct content)
2. Interaction tests (forms, buttons, navigation affected by the diff)
3. Visual regression (compare ARIA snapshots if saved)
4. Accessibility (axe-core scan on affected pages)
"""
If --flow specified, load saved flow from .expect/flows/{slug}.yaml instead of generating.
If NOT -y, present plan to user via AskUserQuestion for review before executing.
Load: Read("references/test-plan.md")
Phase 5: Execution
agent-browser Quick Primer
Floor is
>= 0.31.1(0.27.1 is documented broken on prod pages); current tested release is 0.38.1 (seeupstream-version-tested). Commands below hold across this range. 0.30+ addsagent-browser read(agent-readable text extraction) and the--restore/--namespacesession-restore workflow for stable, isolated browser state across agent runs. 0.33.0 addsagent-browser a11y [url], an embedded axe-core audit (WCAG tag filtering, selector scoping, iframe-aware text/JSON output) available as both a CLI command and an MCP tool. 0.34.0 adds persistent session-to-tab binding for shared Chrome sessions, with--pin-tabmaking the binding strict so an externally closed tab yields a stabletab_goneerror. 0.35.0 adds--ca-cert <path>for a trusted local CA behind an SSL-inspecting proxy (Linux-only; the targeted alternative to--ignore-https-errors). 0.35.1 changes behaviour the Diff row below relies on:diff snapshotnow resets ref numbering per diff and invalidates refs across navigations, so refs captured before a navigation must be re-read rather than reused. 0.36.0 adds experimental WebMCP tool discovery (webmcp list/invoke/result/cancel, brief untrusted catalog summaries onopen). 0.37.0 makesrecordcapture the active page at 30 fps and inherits session setup into new tabs. 0.38.0 addssnapshot --delta(full baseline, thenunchangedor a compact structural change),screenshot --if-changed --threshold, persistent refs that survive same-document DOM changes,--human/--input-modepointer realism, andauth login --no-navigate. Full command surface:../browser-tools/references/upstream.mdandagent-browser skills get core --full.
| Area | Command | Notes |
|---|---|---|
| Snapshot | agent-browser snapshot -i | ARIA tree w/ @eN refs. -C/--cursor was removed in 0.22 |
| Snapshot delta (0.38+) | agent-browser snapshot --delta | Full tree once, then unchanged or a compact structural diff; refs persist across same-document changes |
| Semantic locator | agent-browser find role button click --name "Continue" | Grammar: find <locator> <value> [action]; stable alternative to @eN refs |
| Interaction | fill @e1 "...", click @e2, press Enter, drag @e1 @e2, upload @e1 file.pdf | All take ARIA refs. Add --human for a curved approach instead of an instant jump |
| Waits | wait --load networkidle, wait --text "Success", wait --fn "window.ready" | Event-driven, never sleep-based |
| Network | network route "*analytics*" --abort, network route "https://api/*" --body '{...}' | Intercept + stub |
| State | state save/load auth.json, --session-name <name> | Persist auth across runs |
| Vault | vault store github_pat, vault load github_pat | Encrypted credential store |
| Diff | diff snapshot, diff screenshot --baseline /tmp/x.png | ARIA + pixel diffing across separate runs; prefer snapshot --delta for within-run before/after on the same page |
| Capture | screenshot --annotate, screenshot --if-changed --threshold <n> (0.38+), pdf, record start/stop --fps <n> | --if-changed skips re-encoding an unchanged screenshot; record defaults to 30 fps (0.37+) |
| Dashboard | agent-browser dashboard start (0.25+) | Browser-side runtime inspector on :4848 |
Run the test plan
expect_task = Agent(
subagent_type="ork:expect-agent",
prompt=f"""Execute this test plan:
{test_plan}
For each step:
1. Navigate to the URL
2. Execute the test action
3. Take a screenshot on failure
4. Report PASS/FAIL with evidence
""",
run_in_background=True,
model="sonnet",
max_turns=50
)
# Stream agent-browser progress line-by-line instead of polling
# Each stdout line from agent-browser arrives as a notification — useful for
# catching a failing step early rather than waiting for the full plan.
# Full pattern: Read("${CLAUDE_PLUGIN_ROOT}/skills/chain-patterns/references/monitor-patterns.md")
Monitor(pid=expect_task.agent_id)
# For long test plans (>3 min typical), notify on completion — requires
# Remote Control + "Push when Claude decides" config.
# Skip silently if the user doesn't have Remote Control enabled.
if test_plan_duration_estimate > 180:
PushNotification(
message=f"ork:expect complete — {passed}/{total} steps passed on {len(affected_urls)} pages",
status="proactive"
)
Load: Read("references/execution.md")
Phase 6: Report
expect Report
═══════════════════════════════════════
Target: unstaged (3 files changed)
Pages tested: 4
Duration: 45s
Results:
✓ /login — form renders, submit works
✓ /signup — validation triggers on empty fields
✗ /dashboard — chart component crashes (TypeError)
✓ /settings — preferences save correctly
3 passed, 1 failed
Artifacts:
.expect/reports/2026-03-26T16-30-00.json
.expect/screenshots/dashboard-error.png
Load: Read("references/report.md")
Saved Flows
Reusable test sequences stored as Markdown files with YAML frontmatter in .expect/flows/
(full format: references/saved-flows.md):
---
format_version: 1
title: "Login flow"
slug: "login"
target_scope: "branch"
steps:
- instruction: "Navigate to /login"
expected: "Login form visible with email and password fields"
- instruction: "Fill email and password, click Login"
expected: "Redirect to /dashboard showing 'Welcome back'"
---
Run with: expect --flow login
Auto-trigger after UI edits (M125 #2)
When the dev stack is live (dev), saving any .tsx, .jsx, .css, or .scss file (and Next.js route files like app/**/page.tsx, pages/**/*.tsx) emits a nudge to run expect <route>. The hook (posttool/ui-change-detector) is default-on and:
- skips silently if
devhasn't booted (no agent-browser session to attach to); - enforces a 30-second cooldown per route to prevent spam on rapid saves;
- honors
.claude/state/expect-skip.<sessionId>as a per-session opt-out (write any content); - honors
ORK_EXPECT_AUTO=0for an env-level kill switch.
Route resolution: app/dashboard/page.tsx → /dashboard, pages/settings.tsx → /settings, component / global-style edits → / (home as proxy). Route groups like app/(marketing)/pricing/page.tsx strip to /pricing.
ARIA snapshot recording (M125 #6)
After a passing run, the posttool/expect/snapshot-recorder hook persists the captured ARIA tree to .claude/state/expect-snapshots/<route-slug>/<parent-commit>.json. Subsequent expect <route> --diff runs compare against the most recent prior snapshot for that route — surfaces structural regressions (added/removed buttons, label changes, hierarchy shifts) without needing a baseline screenshot.
For the snapshot recorder to fire, the expect run output must contain RUN_COMPLETED|passed, ROUTE|<route>, and ARIA|<json-summary> tags. The agent-browser-driven flow already emits these.
Jev step judge (opt-in, three modes)
ORK_EXPECT_JEV selects the mode. Unset or falsey is off, and off means zero network. shadow (also selected by a truthy ORK_EXPECT_JEV_SHADOW for compatibility) logs one Jev request per step beside the agent's pick: a Choice over the bounded legal-action set built from the interactive ARIA elements, plus three verify Nouls, with probabilities and per-step agreement in the report. 1 or act lets the Jev pick drive the step instead, failing closed to the agent's own pick on any error, timeout, empty or malformed answer, or a confidence below act_confidence_floor. Tunables live in .expect/config.yaml jev_shadow:. Load: Read("references/jev-shadow.md")
When NOT to Use
- Unit tests — use
coverinstead - API-only changes — no browser UI to test
- Generated files — skip build artifacts, lock files
- Docs-only changes — unless you want to verify docs site rendering
Quality Bar
Done means all of these hold:
- Test-plan scope is derived from the git diff for the chosen
--target— no unaffected page appears in the plan. - Every changed file maps to at least one tested route (via
.expect/config.yamlor inferred convention) OR is explicitly excluded as API-only, generated, or docs-only. - Fingerprint check runs first; a diff unchanged since the last run skips execution instead of re-testing.
- Unless
-yis passed, the plan is presented for review before any browser action runs. - Each executed step reports PASS or FAIL with evidence (screenshot on failure), and the report's pass/fail totals match the steps actually run.
- The report's exit code is non-zero whenever any step failed.
Budget: at most 30 minutes total and 5 minutes per page (a slower page is skipped, per references/test-plan.md; defaults, .expect/config.yaml overrides), 50 executor turns (max_turns=50), one sequential ork:expect-agent, never parallel browsers (rules/no-parallel-browsers.md); stop and report partial results at the finish line or the first cap, whichever comes first.
Related Skills
agent-browser— Browser automation engine (required dependency)ork:cover— Test suite generation (unit/integration/e2e)ork:verify— Grade existing test qualitytesting-e2e— Playwright patterns and best practices
References
Load on demand with Read("references/<file>"):
| File | Content |
|---|---|
fingerprint.md | SHA-256 gating logic |
diff-scanner.md | Git diff parsing + 3-level classification |
route-map.md | File-to-URL mapping conventions |
test-plan.md | AI test plan generation prompt templates |
execution.md | agent-browser orchestration patterns |
report.md | Report format + artifact storage |
config-schema.md | .expect/config.yaml full schema |
aria-diffing.md | ARIA snapshot comparison for semantic diffing |
scope-strategy.md | Test depth strategy per target mode |
saved-flows.md | Markdown+YAML flow format, adaptive replay |
rrweb-recording.md | rrweb DOM replay integration |
human-review.md | AskUserQuestion plan review gate |
ci-integration.md | GitHub Actions workflow + pre-push hooks |
research.md | millionco/expect architecture analysis |
jev-shadow.md | Opt-in Jev step judge per step: bounded-action Choice + verify Nouls; shadow logs, act drives with fail-closed fallback |
Version: 1.2.0 (frontmatter metadata.version is the source of truth). First scaffold March 2026, M99 milestone.
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/yonatangross/orchestkit/expect">View expect on skillZs</a>