octocode-graph-eval
Use when you need a measurable keep/discard loop — goal→KPI, baseline vs target, held-out checks, eval suites, or don't-stop-till-done against a runnable sensor. Not for ordinary ship checks where 'tests passed' is enough.
How do I install this agent skill?
npx skills add https://github.com/bgauryy/octocode --skill octocode-graph-evalIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill is an evaluation framework for AI agents that includes scripts for automated report validation and grading. It presents a low security risk primarily due to its reliance on processing external execution traces and datasets which could contain indirect prompt injections targeting its 'LLM Judge' functionality. Additionally, it encourages autonomous loops without human intervention by default.
- Socketpass
No alerts
- Snykwarn
Risk: MEDIUM · 1 issue
What does this agent skill do?
Octocode Graph Eval
Evaluate outcomes and run improvement loops with evidence, not vibes — for one loop or a graph of loops.
Flow: ERROR-ANALYZE → FRAME(goal→KPI) → BASELINE → LOOP → JUDGE → CAPTURE → VERIFY → SUITE-EVOLVE.
Modes: ErrorAnalyze · Define · Run · Suite · Benchmark · Audit.
Lobby rules
- No goal→KPI link → STOP. No measurable primary → STOP. No runnable sensor → build one before looping.
- Narrative-only accept → REJECT. Editing harness/cases/graders to pass → REJECT.
- ACCEPT only if primary moves on held-out and guardrails hold.
- Prefer deterministic graders; binary/LLM next; humans calibrate. Grade outcomes over paths.
- TDD for agents: write or select a failing case / KPI check before mutating the subject; green only after the change (red → green → keep|discard).
- Public benches orient; private failure suites gate ships. Distrust saturated/contaminated boards.
- Freeze the harness during an experiment; evolve the suite only between experiments.
- Graph check: before evaluating a multi-agent workflow, run edge detection — if no two nodes are independent (every step reads the prior step's output), it is a loop, not a graph. Build a loop.
- Goodhart guard: every primary KPI must have a counter-metric guardrail the agent cannot tune. Primary improving + guardrail degrading → reframe the goal, not the loop.
- Verifier independence: a verifier sharing the executor's context is not independent. Require fresh context before calling a result verified.
- Anchor requirement: every graph must have at least one node whose output cannot be argued with (tests that ran, build exit codes, type errors). No anchors → build one before trusting the graph.
Workflow
- Error-analyze traces into a failure taxonomy; frame success, primary/leading metrics, guardrails, and decision rule.
- Measure a fixed-budget baseline; make the smallest subject change; keep or discard from comparable results.
- Judge grader quality, fairness, capability versus regression, and contamination; capture one durable lesson.
- Verify held-out results and required checks; then add new failure cases between experiments. Stop when goal/KPI is undefined, checks did not run, the harness changed to pass, or another loop cannot change the verdict.
Smart routes — load only what the current step needs
- When deriving failures, load
references/error-analysis.md; when connecting intent to measures loadreferences/goal-kpi-cascade.md, then fillreferences/kpi-contract.md— make success and budget explicit. - When choosing experiment, suite, or meta scope, load
references/nested-loops.md; before the first iteration loadreferences/feedback-loops.md, then for the inner keep/discard cycle loadreferences/agent-loop.md— no workable sensor, no loop. - When the subject is a multi-agent workflow (graph of loops), load
references/graph-of-loops.md— run edge detection first, require anchor nodes, check verifier independence, name Goodhart guardrails, then set primary KPI at the graph boundary with per-node sensors. - When auditing that graph for structural failure risk before trusting its green lights — shared context, opaque state, no checkpoint/resume, unbounded tool permissions, missing human gates — load
references/graph-failure-modes.md; add a suite case on a mode's first trace appearance. - When managing or measuring subagents under eval, load
references/subagent-cookbook.mdfirst for the ownership split; spawn mechanics stay inoctocode-subagent. - When running an evaluated multi-agent iteration, load
references/subagent-protocol.mdfor the frozen FRAME→verdict protocol; when choosing worker and graph-boundary metrics, loadreferences/subagent-kpis.mdso spawn cost is measured, not invisible. - When defining how parent and workers talk during an evaluated run, load
references/subagent-communication.md— bad channels create false certainty and unattributable failures; when choosing the topology itself, loadreferences/subagent-approaches.mdbecause the pattern decides which KPIs and checks matter. - When inner loop is flat and no new hypothesis exists, suspect stuck search priors — load
references/nested-loops.mdfor bilevel escalation, thenreferences/karpathy-patterns.mdfor the Bilevel Autoresearch pattern. - When selecting graders or statistical checks, load
references/eval-techniques.md; when grading agent tool-call sequences or multi-turn trajectories loadreferences/trajectory-grading.md; when trusting public/private suites loadreferences/benchmarking.md— match evidence strength to the decision. - When creating cases and runners, load
references/eval-harness.md; before acceptance loadreferences/held-out-and-guards.md— prevent leakage, overfitting, and greenwashing. - When grounding methods in primary patterns, load
references/karpathy-patterns.md— anchor techniques in proven loops. - When a result needs another skill or durable capture, load
references/routing.md; when closing a meta improvement cycle loadreferences/improve-loop.md— transfer ownership without losing the decision rule. - When reporting, load
references/output.mdand runscripts/loop-report.mjs— require goal, baseline, result, and verdict.
Related routes and verification
- Use
octocode-researchfor evidence under test;octocode-brainstormingbefore evaluating an unresolved idea;octocode-rfc-generatorfor a design KPI contract. - Use
octocode-subagentto fan out parallel hypotheses or benchmark trials within one iteration — measurement, keep/discard, graders, and the subagent cookbook (references/subagent-cookbook.md) stay frozen here. - Use
octocode-prompt-optimizerfor wording after the KPI is fixed;octocode-skillsfor folder edits after ACCEPT. - When changing this skill, run
scripts/check-description.mjsthenscripts/eval-eval.mjs --self-testand a matching--case— catch trigger and self-routing regressions; cases live inevals/(cases.json,trigger-cases.json,kpi-contract.json).
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/bgauryy/octocode/octocode-graph-eval">View octocode-graph-eval on skillZs</a>