skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
yonatangross/orchestkit97 installs

bare-eval

Isolated eval and grading calls with claude -p in bare mode, free of plugin and hook interference. Use when grading skill outputs, benchmarking prompts or testing trigger accuracy.

How do I install this agent skill?

npx skills add https://github.com/yonatangross/orchestkit --skill bare-eval
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubwarn

    The skill provides a framework for isolated evaluation and grading using Claude Code's bare mode. It includes a workflow script for assessing other skills that lacks explicit validation for input parameters used in file path construction, creating a potential path traversal risk.

  • Socketpass

    No alerts

  • Snykpass

    Risk: LOW · No issues

What does this agent skill do?

Bare Eval — Isolated Evaluation Calls

Run claude -p --bare for fast, clean eval/grading without plugin overhead.

CC 2.1.81 required. The --bare flag skips hooks, LSP, plugin sync, and skill directory walks.

When to Use

  • Grading skill outputs against assertions
  • Trigger classification (which skill matches a prompt)
  • Description optimization iterations
  • Any scripted -p call that doesn't need plugins

When NOT to Use

  • Testing skill routing (needs --plugin-dir)
  • Testing agent orchestration (needs full plugin context)
  • Interactive sessions

Prerequisites

# --bare requires ANTHROPIC_API_KEY (OAuth/keychain disabled)
export ANTHROPIC_API_KEY="sk-ant-..."

# Verify CC version
claude --version  # Must be >= 2.1.81

Quick Reference

Call TypeCommand Pattern
Gradingclaude -p "$prompt" --bare --max-turns 1 --output-format text
Triggerclaude -p "$prompt" --bare --json-schema "$schema" --output-format json
Streaming gradeclaude -p "$prompt" --bare --max-turns 1 --output-format stream-json
Optimizeecho "$prompt" | claude -p --bare --max-turns 1 --output-format text
Force-skillclaude -p "$prompt" --bare --print --append-system-prompt "$content"
@-file in promptclaude -p "grade @fixtures/case-1.md against rubric" --bare (CC 2.1.113 Remote Control autocomplete)

Long harness runs (CC 2.1.199+): set CLAUDE_CODE_RETRY_WATCHDOG=1 for unattended eval batches: it raises the default retry count for non-capacity transient errors to 300 and lifts the cap of 15 on CLAUDE_CODE_MAX_RETRIES, so an overnight grading run survives transient API blips instead of dying mid-batch. Pair it with the three caps so a batch neither waits out a 429 storm nor sits hours on a usage limit: CLAUDE_CODE_RETRY_WATCHDOG_MAX_WAIT_MS (2.1.295), CLAUDE_CODE_RETRY_WATCHDOG_MAX_USAGE_LIMIT_WAIT_MS (2.1.301) and CLAUDE_CODE_RETRY_WATCHDOG_USAGE_LIMIT_RECHECK_MS (2.1.305); doctor's operator-permissions check offers any that are missing.

--output-format stream-json

Newline-delimited JSON events (one per token/tool-call) — lets a runner score partial output or abort early on a failing probe without waiting for the full response.

claude -p "$prompt" --bare --max-turns 1 --output-format stream-json \
  | while IFS= read -r line; do
      # line is a single JSON event; inspect $.type == "content_block_delta"
      jq -r 'select(.type == "content_block_delta") | .delta.text' <<< "$line"
    done

Use stream-json over json when:

  • grading long outputs and you want incremental scoring,
  • piping into another CLI step-by-step (e.g. ork:eval-runner),
  • you need per-token timing data alongside the content.

Invocation Patterns

Load detailed patterns and examples:

Read("references/invocation-patterns.md")

Grading Schemas

JSON schemas for structured eval output:

Read("references/grading-schemas.md")

Pipeline Integration

OrchestKit's eval scripts (npm run eval:skill) auto-detect bare mode:

# eval-common.sh detects ANTHROPIC_API_KEY → sets BARE_MODE=true
# Scripts add --bare to all non-plugin calls automatically

Bare calls: Trigger classification, force-skill, baseline, all grading. Never bare: run_with_skill (needs plugin context for routing tests).

CC 2.1.119: --print honors agent tools: / disallowedTools: (M122)

Before CC 2.1.119, --print mode ran with the full default tool set regardless of the agent's frontmatter tools: and disallowedTools:. Bare-eval grading was effectively ungated — graders could call any tool they wanted, even if the agent definition restricted them.

As of 2.1.119, --print enforces the agent's declared tool surface. Implications for eval design:

ConsequenceAction
Eval graders that relied on unrestricted tool access may now failAudit grader prompts for tools they actually need; whitelist explicitly via the agent's tools: frontmatter
Eval results match interactive runsReproducibility improves — grading what the model can actually do, not what it could do in an unsandboxed --print
--agent <name> also honors permissionMode in --printPermission-gated tools (Bash, Edit) require either permissionMode: acceptEdits or explicit allowlists in the agent definition

Migration test:

# Run an eval against an agent with a deliberately tight tools: list.
# Graders that previously called Read/Bash freely will now fail unless those
# tools are declared on the agent.
claude -p "$prompt" --bare --print --agent grader-test

If the grader fails with a "tool not permitted" error, add the required tool to the agent's tools: frontmatter and re-run.

CC 2.1.121: CLAUDE_CODE_FORK_SUBAGENT=1 for grader determinism (#1545)

Before CC 2.1.121, the env var only worked in interactive sessions. As of 2.1.121, non-interactive paths (claude -p, SDK) honor it too — each grader invocation gets a fresh forked subagent context.

The cross-eval state-leak problem this fixes:

Without forking, sequential claude -p --bare graders inherit harness state:

InheritedSymptom
memory MCP query cachegrader sees stale hit from previous run; same fixture grades differently
.claude/chain/*.json on diskgrader for "implement" thinks "explore" already ran (file is from previous test)
ToolSearch deferred-tool cachefirst grader's MCP loads bleed into next grader's tool registry
model picker prefgrader N inherits --model=opus from grader N-1

This produced ~5–10% retry rate and non-reproducible scores — the eval baseline drifted between runs, engineers chased phantom regressions.

Fix: tests/evals/scripts/lib/eval-common.sh exports CLAUDE_CODE_FORK_SUBAGENT=1, so every script that sources it (run-trigger-eval, run-quality-eval, run-agent-eval, optimize-description, etc.) gets forked graders automatically. The CI workflow .github/workflows/skill-eval.yml also sets it at the workflow level (its predecessor orchestkit-eval.yml was retired 2026-08-01 — its grading phases rated description prose, not behavior). Older CC silently ignores the env var (no-op).

Determinism contract: running the same grader on the same fixture twice in a row produces the same score. Verified by tests/evals/scripts/test-grader-determinism.sh.

Performance

ScenarioWithout --bareWith --bareSavings
Single grading call~3-5s startup~0.5-1s2-4x
Trigger (per prompt)~3-5s~0.5-1s2-4x
Full eval (50 calls)~150-250s overhead~25-50s3-5x

Rules

Read("rules/_sections.md")

Troubleshooting

Read("references/troubleshooting.md")

Dynamic-workflow harness (template-in-skill)

workflows/skill-fitness.js is a runnable dynamic-workflow template — the workflow-backed complement to the static conformance grader (scripts/eval/conformance-check.mjs). It fans out one isolated-context agent per skill to score fitness (freshness / router-clarity / structure) and synthesizes a ranked scorecard, catching qualitative drift a static grep can't (description/body count mismatches, duplicate headings, install-specific absolute paths, version drift). Run it with the Workflow tool:

Workflow({ scriptPath: "${CLAUDE_SKILL_DIR}/workflows/skill-fitness.js",
           args: ["assess", "commit", "doctor"] })

Treat it as a template, not a verbatim script — adapt the SKILLS list and rubric per use. Cost is real (~50k tokens/skill; scoring all ~112 is ~6M tokens), so pass an explicit batch via args. Static-first: run conformance-check.mjs (zero tokens) to pre-filter, then this harness for the judgment grep can't make.

Holdout Bake-Off Grading

The holdout-promotion gate grades a champion and a challenger SKILL.md over the same sealed holdout via bare-mode forked graders — the canonical consumer of the determinism contract above: identical grader + identical ork-rubric/1.0 + identical sealed set, with CLAUDE_CODE_FORK_SUBAGENT=1 so the only variable is the version under test. Both --bare constraints apply (requires ANTHROPIC_API_KEY, bills tokens directly → on-demand / CI only). Run it with bash tests/evals/scripts/run-skill-eval.sh --holdout-promote <skill>.

Related

  • eval:skill npm script — unified skill evaluation runner
  • eval:trigger — trigger accuracy testing
  • eval:quality — A/B quality comparison
  • optimize-description.sh — iterative description improvement
  • Version compatibility: doctor/references/version-compatibility.md

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/yonatangross/orchestkit/bare-eval">View bare-eval on skillZs</a>