skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
yonatangross/orchestkit92 installs

expect

Diff-aware browser testing: maps the git diff to affected pages, plans and runs targeted checks via agent-browser. Use when testing UI changes before merge. Not for writing tests (use cover), grading a suite (use verify) or unit tests.

How do I install this agent skill?

npx skills add https://github.com/yonatangross/orchestkit --skill expect
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    This skill provides a comprehensive browser testing framework using git diffs to target affected UI components. It includes an optional AI-driven judging feature that communicates with an external API and supports session recording via script injection.

  • Socketwarn

    2 alerts: gptAnomaly

  • Snykwarn

    Risk: MEDIUM · 2 issues

What does this agent skill do?

Expect — Diff-Aware AI Browser Testing

Analyze git changes, generate targeted test plans, and execute them via AI-driven browser automation.

Note: If disableSkillShellExecution is enabled, the agent-browser install check won't run. Verify it's installed: npx agent-browser --version.

expect                              # Auto-detect changes, test affected pages
expect -m "test the checkout flow"  # Specific instruction
expect --flow login                 # Replay a saved test flow
expect --target branch              # Test all changes on current branch vs main
expect -y                           # Skip plan review, run immediately

Core principle: Only test what changed. Git diff drives scope — no wasted cycles on unaffected pages.

Argument Resolution

ARGS = "[-m <instruction>] [--target unstaged|branch|commit] [--flow <slug>] [-y]"

# Parse from full argument string
import re
raw = "$ARGUMENTS"  # Full argument string

INSTRUCTION = None
TARGET = "unstaged"  # Default: test unstaged changes
FLOW = None
SKIP_REVIEW = False

# Extract -m "instruction"
m_match = re.search(r'-m\s+["\']([^"\']+)["\']|-m\s+(\S+)', raw)
if m_match:
    INSTRUCTION = m_match.group(1) or m_match.group(2)

# Extract --target
t_match = re.search(r'--target\s+(unstaged|branch|commit)', raw)
if t_match:
    TARGET = t_match.group(1)

# Extract --flow
f_match = re.search(r'--flow\s+(\S+)', raw)
if f_match:
    FLOW = f_match.group(1)

# Extract -y
if '-y' in raw.split():
    SKIP_REVIEW = True

STEP 0: MCP Probe + Prerequisite Check

# memory may be absent from the user's .mcp.json; probe so the memory steps below are skipped, not failed:
ToolSearch(query="select:mcp__memory__search_nodes")

# Verify agent-browser is available (Rust-native, no Playwright)
Bash("command -v agent-browser || npx agent-browser --version")
# If missing: "Install agent-browser: npm i -g agent-browser"

# Load agent-browser's version-matched workflow guide (ships with the CLI).
# NOT `skills get agent-browser` — that resolves but returns only the thin
# top-level router; `core --full` is the actual 2,800+ line command reference.
Bash("agent-browser skills get core --full")

CRITICAL: Task Management

# 1. Create main task IMMEDIATELY
TaskCreate(
  subject="Expect: test changed code",
  description="Diff-aware browser testing pipeline",
  activeForm="Running diff-aware browser tests"
)

# 2. Create subtasks for each pipeline phase
TaskCreate(subject="Check fingerprint (skip if unchanged)", activeForm="Checking fingerprint")  # id=2
TaskCreate(subject="Scan git diff and classify changes", activeForm="Scanning diff")            # id=3
TaskCreate(subject="Map changes to routes/URLs", activeForm="Mapping routes")                   # id=4
TaskCreate(subject="Generate AI test plan", activeForm="Generating test plan")                   # id=5
TaskCreate(subject="Execute tests via agent-browser", activeForm="Executing browser tests")     # id=6
TaskCreate(subject="Compile test report", activeForm="Compiling report")                        # id=7

# 3. Set dependencies for sequential phases
TaskUpdate(taskId="3", addBlockedBy=["2"])  # Diff scan needs fingerprint check
TaskUpdate(taskId="4", addBlockedBy=["3"])  # Route map needs diff results
TaskUpdate(taskId="5", addBlockedBy=["4"])  # Test plan needs route map
TaskUpdate(taskId="6", addBlockedBy=["5"])  # Execution needs test plan
TaskUpdate(taskId="7", addBlockedBy=["6"])  # Report needs execution results

# 4. Update status as you progress
TaskUpdate(taskId="2", status="in_progress")  # When starting
TaskUpdate(taskId="2", status="completed")    # When done — repeat for each subtask

Pipeline Overview

Git Diff → Route Map → Fingerprint Check → Test Plan → Execute → Report
PhaseWhatOutputReference
1. FingerprintSHA-256 hash of changed filesSkip if unchanged since last runreferences/fingerprint.md
2. Diff ScanParse git diff, classify changesChangesFor data (files, components, routes)references/diff-scanner.md
3. Route MapMap changed files to affected pages/URLsScoped page listreferences/route-map.md
4. Test PlanGenerate AI test plan from diff + route mapMarkdown test plan with stepsreferences/test-plan.md
5. ExecuteRun test plan via agent-browserPass/fail per step, screenshotsreferences/execution.md
6. ReportAggregate results, artifacts, exit codeStructured report + artifactsreferences/report.md

Phase 1: Fingerprint Check

Check if the current changes have already been tested:

Read(".expect/fingerprints.json")  # Previous run hashes
# Compare SHA-256 of changed files against stored fingerprints
# If match: "No changes since last test run. Use --force to re-run."
# If no match or --force: continue to Phase 2

Load: Read("references/fingerprint.md")

Phase 2: Diff Scan

Analyze git changes based on --target:

if TARGET == "unstaged":
    diff = Bash("git diff")
    files = Bash("git diff --name-only")
elif TARGET == "branch":
    diff = Bash("git diff main...HEAD")
    files = Bash("git diff main...HEAD --name-only")
elif TARGET == "commit":
    diff = Bash("git diff HEAD~1")
    files = Bash("git diff HEAD~1 --name-only")

Classify each changed file into 3 levels:

  1. Direct — the file itself changed
  2. Imported — a file that imports the changed file
  3. Routed — the page/route that renders the changed component

Load: Read("references/diff-scanner.md")

Phase 3: Route Map

Map changed files to testable URLs using .expect/config.yaml:

# .expect/config.yaml
base_url: http://localhost:3000
route_map:
  "src/components/Header.tsx": ["/", "/about", "/pricing"]
  "src/app/auth/**": ["/login", "/signup", "/forgot-password"]
  "src/app/dashboard/**": ["/dashboard"]

If no route map exists, infer from Next.js App Router / Pages Router conventions.

Load: Read("references/route-map.md")

Phase 4: Test Plan Generation

Build an AI test plan scoped to the diff, using the scope strategy for the current target:

scope_strategy = get_scope_strategy(TARGET)  # See references/scope-strategy.md

prompt = f"""
{scope_strategy}

Changes: {diff_summary}
Affected pages: {affected_urls}
Instruction: {INSTRUCTION or "Test that the changes work correctly"}

Generate a test plan with:
1. Page-level checks (loads, no console errors, correct content)
2. Interaction tests (forms, buttons, navigation affected by the diff)
3. Visual regression (compare ARIA snapshots if saved)
4. Accessibility (axe-core scan on affected pages)
"""

If --flow specified, load saved flow from .expect/flows/{slug}.yaml instead of generating.

If NOT -y, present plan to user via AskUserQuestion for review before executing.

Load: Read("references/test-plan.md")

Phase 5: Execution

agent-browser Quick Primer

Floor is >= 0.31.1 (0.27.1 is documented broken on prod pages); current tested release is 0.38.1 (see upstream-version-tested). Commands below hold across this range. 0.30+ adds agent-browser read (agent-readable text extraction) and the --restore / --namespace session-restore workflow for stable, isolated browser state across agent runs. 0.33.0 adds agent-browser a11y [url], an embedded axe-core audit (WCAG tag filtering, selector scoping, iframe-aware text/JSON output) available as both a CLI command and an MCP tool. 0.34.0 adds persistent session-to-tab binding for shared Chrome sessions, with --pin-tab making the binding strict so an externally closed tab yields a stable tab_gone error. 0.35.0 adds --ca-cert <path> for a trusted local CA behind an SSL-inspecting proxy (Linux-only; the targeted alternative to --ignore-https-errors). 0.35.1 changes behaviour the Diff row below relies on: diff snapshot now resets ref numbering per diff and invalidates refs across navigations, so refs captured before a navigation must be re-read rather than reused. 0.36.0 adds experimental WebMCP tool discovery (webmcp list/invoke/result/cancel, brief untrusted catalog summaries on open). 0.37.0 makes record capture the active page at 30 fps and inherits session setup into new tabs. 0.38.0 adds snapshot --delta (full baseline, then unchanged or a compact structural change), screenshot --if-changed --threshold, persistent refs that survive same-document DOM changes, --human / --input-mode pointer realism, and auth login --no-navigate. Full command surface: ../browser-tools/references/upstream.md and agent-browser skills get core --full.

AreaCommandNotes
Snapshotagent-browser snapshot -iARIA tree w/ @eN refs. -C/--cursor was removed in 0.22
Snapshot delta (0.38+)agent-browser snapshot --deltaFull tree once, then unchanged or a compact structural diff; refs persist across same-document changes
Semantic locatoragent-browser find role button click --name "Continue"Grammar: find <locator> <value> [action]; stable alternative to @eN refs
Interactionfill @e1 "...", click @e2, press Enter, drag @e1 @e2, upload @e1 file.pdfAll take ARIA refs. Add --human for a curved approach instead of an instant jump
Waitswait --load networkidle, wait --text "Success", wait --fn "window.ready"Event-driven, never sleep-based
Networknetwork route "*analytics*" --abort, network route "https://api/*" --body '{...}'Intercept + stub
Statestate save/load auth.json, --session-name <name>Persist auth across runs
Vaultvault store github_pat, vault load github_patEncrypted credential store
Diffdiff snapshot, diff screenshot --baseline /tmp/x.pngARIA + pixel diffing across separate runs; prefer snapshot --delta for within-run before/after on the same page
Capturescreenshot --annotate, screenshot --if-changed --threshold <n> (0.38+), pdf, record start/stop --fps <n>--if-changed skips re-encoding an unchanged screenshot; record defaults to 30 fps (0.37+)
Dashboardagent-browser dashboard start (0.25+)Browser-side runtime inspector on :4848

Run the test plan

expect_task = Agent(
  subagent_type="ork:expect-agent",
  prompt=f"""Execute this test plan:
  {test_plan}

  For each step:
  1. Navigate to the URL
  2. Execute the test action
  3. Take a screenshot on failure
  4. Report PASS/FAIL with evidence
  """,
  run_in_background=True,
  model="sonnet",
  max_turns=50
)

# Stream agent-browser progress line-by-line instead of polling
# Each stdout line from agent-browser arrives as a notification — useful for
# catching a failing step early rather than waiting for the full plan.
# Full pattern: Read("${CLAUDE_PLUGIN_ROOT}/skills/chain-patterns/references/monitor-patterns.md")
Monitor(pid=expect_task.agent_id)

# For long test plans (>3 min typical), notify on completion — requires
# Remote Control + "Push when Claude decides" config.
# Skip silently if the user doesn't have Remote Control enabled.
if test_plan_duration_estimate > 180:
    PushNotification(
        message=f"ork:expect complete — {passed}/{total} steps passed on {len(affected_urls)} pages",
        status="proactive"
    )

Load: Read("references/execution.md")

Phase 6: Report

expect Report
═══════════════════════════════════════
Target: unstaged (3 files changed)
Pages tested: 4
Duration: 45s

Results:
  ✓ /login — form renders, submit works
  ✓ /signup — validation triggers on empty fields
  ✗ /dashboard — chart component crashes (TypeError)
  ✓ /settings — preferences save correctly

3 passed, 1 failed

Artifacts:
  .expect/reports/2026-03-26T16-30-00.json
  .expect/screenshots/dashboard-error.png

Load: Read("references/report.md")

Saved Flows

Reusable test sequences stored as Markdown files with YAML frontmatter in .expect/flows/ (full format: references/saved-flows.md):

---
format_version: 1
title: "Login flow"
slug: "login"
target_scope: "branch"
steps:
  - instruction: "Navigate to /login"
    expected: "Login form visible with email and password fields"
  - instruction: "Fill email and password, click Login"
    expected: "Redirect to /dashboard showing 'Welcome back'"
---

Run with: expect --flow login

Auto-trigger after UI edits (M125 #2)

When the dev stack is live (dev), saving any .tsx, .jsx, .css, or .scss file (and Next.js route files like app/**/page.tsx, pages/**/*.tsx) emits a nudge to run expect <route>. The hook (posttool/ui-change-detector) is default-on and:

  • skips silently if dev hasn't booted (no agent-browser session to attach to);
  • enforces a 30-second cooldown per route to prevent spam on rapid saves;
  • honors .claude/state/expect-skip.<sessionId> as a per-session opt-out (write any content);
  • honors ORK_EXPECT_AUTO=0 for an env-level kill switch.

Route resolution: app/dashboard/page.tsx → /dashboard, pages/settings.tsx → /settings, component / global-style edits → / (home as proxy). Route groups like app/(marketing)/pricing/page.tsx strip to /pricing.

ARIA snapshot recording (M125 #6)

After a passing run, the posttool/expect/snapshot-recorder hook persists the captured ARIA tree to .claude/state/expect-snapshots/<route-slug>/<parent-commit>.json. Subsequent expect <route> --diff runs compare against the most recent prior snapshot for that route — surfaces structural regressions (added/removed buttons, label changes, hierarchy shifts) without needing a baseline screenshot.

For the snapshot recorder to fire, the expect run output must contain RUN_COMPLETED|passed, ROUTE|<route>, and ARIA|<json-summary> tags. The agent-browser-driven flow already emits these.

Jev step judge (opt-in, three modes)

ORK_EXPECT_JEV selects the mode. Unset or falsey is off, and off means zero network. shadow (also selected by a truthy ORK_EXPECT_JEV_SHADOW for compatibility) logs one Jev request per step beside the agent's pick: a Choice over the bounded legal-action set built from the interactive ARIA elements, plus three verify Nouls, with probabilities and per-step agreement in the report. 1 or act lets the Jev pick drive the step instead, failing closed to the agent's own pick on any error, timeout, empty or malformed answer, or a confidence below act_confidence_floor. Tunables live in .expect/config.yaml jev_shadow:. Load: Read("references/jev-shadow.md")

When NOT to Use

  • Unit tests — use cover instead
  • API-only changes — no browser UI to test
  • Generated files — skip build artifacts, lock files
  • Docs-only changes — unless you want to verify docs site rendering

Quality Bar

Done means all of these hold:

  • Test-plan scope is derived from the git diff for the chosen --target — no unaffected page appears in the plan.
  • Every changed file maps to at least one tested route (via .expect/config.yaml or inferred convention) OR is explicitly excluded as API-only, generated, or docs-only.
  • Fingerprint check runs first; a diff unchanged since the last run skips execution instead of re-testing.
  • Unless -y is passed, the plan is presented for review before any browser action runs.
  • Each executed step reports PASS or FAIL with evidence (screenshot on failure), and the report's pass/fail totals match the steps actually run.
  • The report's exit code is non-zero whenever any step failed.

Budget: at most 30 minutes total and 5 minutes per page (a slower page is skipped, per references/test-plan.md; defaults, .expect/config.yaml overrides), 50 executor turns (max_turns=50), one sequential ork:expect-agent, never parallel browsers (rules/no-parallel-browsers.md); stop and report partial results at the finish line or the first cap, whichever comes first.

Related Skills

  • agent-browser — Browser automation engine (required dependency)
  • ork:cover — Test suite generation (unit/integration/e2e)
  • ork:verify — Grade existing test quality
  • testing-e2e — Playwright patterns and best practices

References

Load on demand with Read("references/<file>"):

FileContent
fingerprint.mdSHA-256 gating logic
diff-scanner.mdGit diff parsing + 3-level classification
route-map.mdFile-to-URL mapping conventions
test-plan.mdAI test plan generation prompt templates
execution.mdagent-browser orchestration patterns
report.mdReport format + artifact storage
config-schema.md.expect/config.yaml full schema
aria-diffing.mdARIA snapshot comparison for semantic diffing
scope-strategy.mdTest depth strategy per target mode
saved-flows.mdMarkdown+YAML flow format, adaptive replay
rrweb-recording.mdrrweb DOM replay integration
human-review.mdAskUserQuestion plan review gate
ci-integration.mdGitHub Actions workflow + pre-push hooks
research.mdmillionco/expect architecture analysis
jev-shadow.mdOpt-in Jev step judge per step: bounded-action Choice + verify Nouls; shadow logs, act drives with fail-closed fallback

Version: 1.2.0 (frontmatter metadata.version is the source of truth). First scaffold March 2026, M99 milestone.

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/yonatangross/orchestkit/expect">View expect on skillZs</a>