skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
memorysaver/agentic-engineering-patterns59 installs

aep-gen-eval

Defines the reusable generator/evaluator pattern: scoring, agent contracts, eval protocol. Validating one specific artifact is /aep-validate.

How do I install this agent skill?

npx skills add https://github.com/memorysaver/agentic-engineering-patterns --skill aep-gen-eval
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubfail

    The skill provides a framework for multi-agent evaluation but contains high-risk instructions to bypass AI agent security permissions and sandboxing. Specifically, the execution protocol directs the agent to spawn sub-processes using flags like '--dangerously-skip-permissions', which explicitly disables security constraints. It also relies on shell-based process control through tmux, which presents a significant attack surface when handling untrusted artifacts.

  • Socketpass

    No alerts

  • Snykpass

    Risk: LOW · No issues

What does this agent skill do?

Generator/Evaluator Pattern

A reusable design pattern for honest evaluation of agent-produced artifacts. Separates the agent that creates work (generator) from the agent that evaluates it (evaluator), because agents consistently praise their own work.

"When asked to evaluate work they've produced, agents tend to respond by confidently praising the work — even when, to a human observer, the quality is obviously mediocre." — Anthropic, "Harness Design for Long-Running Application Development"

Dual-use: consumer skills read this skill's references/ files — the canonical homes for scoring, contracts, protocol, recovery, and findings — while invoking /aep-gen-eval directly runs a full gen/eval loop on any artifact.


How Other Skills Use This

SkillWhat it usesReference files
/aep-build Phase 5Scoring framework + eval protocolscoring-framework.md, eval-protocol.md, recovery-ladder.md
/aep-launchDimension presets for brainstormingscoring-framework.md (presets section)
/aep-validateAgent prompts + findings formatagent-contracts.md, findings-format.md, scoring-framework.md
/aep-build, /aep-dispatch, /aep-wrap, /aep-autopilotFailure taxonomy, verification tiers/recipes, accountingverification-economics.md

The Core Principle

Generator and evaluator are separate agents. Self-evaluation produces inflated scores and rationalized problems (the finding quoted above); an independent evaluator catches what the generator is blind to.

Scaling up: generator/evaluator is the canonical instance of adversarial verification. When one task produces many findings/claims that each need an independent check, /aep-workflow generalizes this to a fan-out of N verifiers/refuters — reusing this skill's scoring framework and findings format per finding.


Reference Files

These files are the canonical homes for the gen/eval contracts every consumer (/aep-build, /aep-validate, /aep-launch) points at. Read the one the branch needs; each file is self-contained.

FileContentsWhen to read
references/scoring-framework.mdDimension definitions (1-5 scale), hard failure thresholds, dimension presets (UI, API, security, data, mixed), few-shot examples, anti-patternsSetting up evaluation criteria, scoring work, calibrating evaluators
references/agent-contracts.mdGenerator/evaluator role separation, prompt templates (generator, evaluator, protocol checker), context assembly rulesSpawning evaluation agents, assembling prompts
references/eval-protocol.mdEval request/response format, verification JSON schema, the eval loop (request → response → fix → re-evaluate), execution contexts (Task subagent, codex exec, tmux, workflow), the needs-human gate recordRunning the evaluation loop, tracking verification state
references/recovery-ladder.mdTwo-round recovery ladder (same fix → re-ground → human gate proposing fresh generator / decompose) for a blocking finding that survives round 1A blocking finding is still open after round 1
references/findings-format.mdSeverity categorization (blocking/important/minor), deduplication protocol, presentation format, changelog entry formatConsolidating findings from multiple agents, presenting results
references/verification-economics.mdValidator placement matrix, failure taxonomy + classification authority, environment preflight, verification tiers + two-point derivation, verification recipe, accounting sensors, tamper-evident evidence classesRouting a FAIL, deriving verification depth, pricing/recording verification spend

Standalone Usage

When invoked directly, this skill runs a gen/eval loop on any artifact.

Step 1: Identify the artifact

What is being evaluated? Options:

  • A document (product context, architecture, design doc)
  • Code changes (implementation, PR diff)
  • An OpenSpec change (proposal, design, specs, tasks)
  • A structured file (YAML, JSON config, migration plan)

Step 2: Choose execution mode

ModeAgentsWhen to use
ParallelGenerator + Evaluator spawned simultaneouslyDocuments, designs, product context — agents work independently
SequentialGenerator first, then Evaluator reads generator's workCode review — evaluator needs to see the implementation
LoopGenerator → Evaluator → fix → re-evaluate (max 2 rounds)Active development — generator can fix issues between rounds

Step 3: Configure dimensions

Read references/scoring-framework.md and select the preset that matches the artifact (UI-heavy, API-only, security-sensitive, data pipeline, mixed/full-stack, product/design, or document), or define custom dimensions. The preset tables, hard-failure thresholds, and few-shot calibration all live in that file.

Step 4: Spawn agents

Read references/agent-contracts.md for prompt templates. Customize the templates with:

  • The artifact content
  • The technical constraints
  • The verification checklist (what the evaluator should check against the codebase)

Step 5: Process results

Read references/findings-format.md to consolidate, categorize, and present findings, then converge to one of two checkable end states:

  • Fixes applied and re-scored: the generator applies the fixes and the artifact passes a fresh evaluation round with no blocking findings remaining. The cap is two rounds; a blocking finding still open after round 2 escalates per references/recovery-ladder.md.
  • Findings handed off: a consolidated findings file is written to a named path (e.g. <artifact-dir>/eval-findings.md) for a downstream owner to act on.

Design Decisions

Gen/eval is packaged as its own invocable skill that doubles as a reference library, rather than folded into /aep-validate or /aep-launch. Rationale: docs/decisions/gen-eval-rationale.md.


Next Step

After running gen/eval, proceed based on what was evaluated:

  • Product context → /aep-dispatch
  • Design artifacts → /aep-launch
  • Code → create PR or continue /aep-build
  • Documents → publish or share

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/memorysaver/agentic-engineering-patterns/aep-gen-eval">View aep-gen-eval on skillZs</a>