bare-eval
Isolated eval and grading calls with claude -p in bare mode, free of plugin and hook interference. Use when grading skill outputs, benchmarking prompts or testing trigger accuracy.
How do I install this agent skill?
npx skills add https://github.com/yonatangross/orchestkit --skill bare-evalIs this agent skill safe to install?
- Gen Agent Trust Hubwarn
The skill provides a framework for isolated evaluation and grading using Claude Code's bare mode. It includes a workflow script for assessing other skills that lacks explicit validation for input parameters used in file path construction, creating a potential path traversal risk.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
What does this agent skill do?
Bare Eval — Isolated Evaluation Calls
Run claude -p --bare for fast, clean eval/grading without plugin overhead.
CC 2.1.81 required. The --bare flag skips hooks, LSP, plugin sync, and skill directory walks.
When to Use
- Grading skill outputs against assertions
- Trigger classification (which skill matches a prompt)
- Description optimization iterations
- Any scripted
-pcall that doesn't need plugins
When NOT to Use
- Testing skill routing (needs
--plugin-dir) - Testing agent orchestration (needs full plugin context)
- Interactive sessions
Prerequisites
# --bare requires ANTHROPIC_API_KEY (OAuth/keychain disabled)
export ANTHROPIC_API_KEY="sk-ant-..."
# Verify CC version
claude --version # Must be >= 2.1.81
Quick Reference
| Call Type | Command Pattern |
|---|---|
| Grading | claude -p "$prompt" --bare --max-turns 1 --output-format text |
| Trigger | claude -p "$prompt" --bare --json-schema "$schema" --output-format json |
| Streaming grade | claude -p "$prompt" --bare --max-turns 1 --output-format stream-json |
| Optimize | echo "$prompt" | claude -p --bare --max-turns 1 --output-format text |
| Force-skill | claude -p "$prompt" --bare --print --append-system-prompt "$content" |
| @-file in prompt | claude -p "grade @fixtures/case-1.md against rubric" --bare (CC 2.1.113 Remote Control autocomplete) |
Long harness runs (CC 2.1.199+): set
CLAUDE_CODE_RETRY_WATCHDOG=1for unattended eval batches: it raises the default retry count for non-capacity transient errors to 300 and lifts the cap of 15 onCLAUDE_CODE_MAX_RETRIES, so an overnight grading run survives transient API blips instead of dying mid-batch. Pair it with the three caps so a batch neither waits out a 429 storm nor sits hours on a usage limit:CLAUDE_CODE_RETRY_WATCHDOG_MAX_WAIT_MS(2.1.295),CLAUDE_CODE_RETRY_WATCHDOG_MAX_USAGE_LIMIT_WAIT_MS(2.1.301) andCLAUDE_CODE_RETRY_WATCHDOG_USAGE_LIMIT_RECHECK_MS(2.1.305);doctor's operator-permissions check offers any that are missing.
--output-format stream-json
Newline-delimited JSON events (one per token/tool-call) — lets a runner score partial output or abort early on a failing probe without waiting for the full response.
claude -p "$prompt" --bare --max-turns 1 --output-format stream-json \
| while IFS= read -r line; do
# line is a single JSON event; inspect $.type == "content_block_delta"
jq -r 'select(.type == "content_block_delta") | .delta.text' <<< "$line"
done
Use stream-json over json when:
- grading long outputs and you want incremental scoring,
- piping into another CLI step-by-step (e.g.
ork:eval-runner), - you need per-token timing data alongside the content.
Invocation Patterns
Load detailed patterns and examples:
Read("references/invocation-patterns.md")
Grading Schemas
JSON schemas for structured eval output:
Read("references/grading-schemas.md")
Pipeline Integration
OrchestKit's eval scripts (npm run eval:skill) auto-detect bare mode:
# eval-common.sh detects ANTHROPIC_API_KEY → sets BARE_MODE=true
# Scripts add --bare to all non-plugin calls automatically
Bare calls: Trigger classification, force-skill, baseline, all grading.
Never bare: run_with_skill (needs plugin context for routing tests).
CC 2.1.119: --print honors agent tools: / disallowedTools: (M122)
Before CC 2.1.119, --print mode ran with the full default tool set regardless of the agent's frontmatter tools: and disallowedTools:. Bare-eval grading was effectively ungated — graders could call any tool they wanted, even if the agent definition restricted them.
As of 2.1.119, --print enforces the agent's declared tool surface. Implications for eval design:
| Consequence | Action |
|---|---|
| Eval graders that relied on unrestricted tool access may now fail | Audit grader prompts for tools they actually need; whitelist explicitly via the agent's tools: frontmatter |
| Eval results match interactive runs | Reproducibility improves — grading what the model can actually do, not what it could do in an unsandboxed --print |
--agent <name> also honors permissionMode in --print | Permission-gated tools (Bash, Edit) require either permissionMode: acceptEdits or explicit allowlists in the agent definition |
Migration test:
# Run an eval against an agent with a deliberately tight tools: list.
# Graders that previously called Read/Bash freely will now fail unless those
# tools are declared on the agent.
claude -p "$prompt" --bare --print --agent grader-test
If the grader fails with a "tool not permitted" error, add the required tool to the agent's tools: frontmatter and re-run.
CC 2.1.121: CLAUDE_CODE_FORK_SUBAGENT=1 for grader determinism (#1545)
Before CC 2.1.121, the env var only worked in interactive sessions. As of 2.1.121, non-interactive paths (claude -p, SDK) honor it too — each grader invocation gets a fresh forked subagent context.
The cross-eval state-leak problem this fixes:
Without forking, sequential claude -p --bare graders inherit harness state:
| Inherited | Symptom |
|---|---|
| memory MCP query cache | grader sees stale hit from previous run; same fixture grades differently |
.claude/chain/*.json on disk | grader for "implement" thinks "explore" already ran (file is from previous test) |
| ToolSearch deferred-tool cache | first grader's MCP loads bleed into next grader's tool registry |
| model picker pref | grader N inherits --model=opus from grader N-1 |
This produced ~5–10% retry rate and non-reproducible scores — the eval baseline drifted between runs, engineers chased phantom regressions.
Fix: tests/evals/scripts/lib/eval-common.sh exports CLAUDE_CODE_FORK_SUBAGENT=1, so every script that sources it (run-trigger-eval, run-quality-eval, run-agent-eval, optimize-description, etc.) gets forked graders automatically. The CI workflow .github/workflows/skill-eval.yml also sets it at the workflow level (its predecessor orchestkit-eval.yml was retired 2026-08-01 — its grading phases rated description prose, not behavior). Older CC silently ignores the env var (no-op).
Determinism contract: running the same grader on the same fixture twice in a row produces the same score. Verified by tests/evals/scripts/test-grader-determinism.sh.
Performance
| Scenario | Without --bare | With --bare | Savings |
|---|---|---|---|
| Single grading call | ~3-5s startup | ~0.5-1s | 2-4x |
| Trigger (per prompt) | ~3-5s | ~0.5-1s | 2-4x |
| Full eval (50 calls) | ~150-250s overhead | ~25-50s | 3-5x |
Rules
Read("rules/_sections.md")
Troubleshooting
Read("references/troubleshooting.md")
Dynamic-workflow harness (template-in-skill)
workflows/skill-fitness.js is a runnable dynamic-workflow template — the workflow-backed complement to the static conformance grader (scripts/eval/conformance-check.mjs). It fans out one isolated-context agent per skill to score fitness (freshness / router-clarity / structure) and synthesizes a ranked scorecard, catching qualitative drift a static grep can't (description/body count mismatches, duplicate headings, install-specific absolute paths, version drift). Run it with the Workflow tool:
Workflow({ scriptPath: "${CLAUDE_SKILL_DIR}/workflows/skill-fitness.js",
args: ["assess", "commit", "doctor"] })
Treat it as a template, not a verbatim script — adapt the SKILLS list and rubric per use. Cost is real (~50k tokens/skill; scoring all ~112 is ~6M tokens), so pass an explicit batch via args. Static-first: run conformance-check.mjs (zero tokens) to pre-filter, then this harness for the judgment grep can't make.
Holdout Bake-Off Grading
The holdout-promotion gate grades a champion and a challenger SKILL.md over the same sealed holdout via bare-mode forked graders — the canonical consumer of the determinism contract above: identical grader + identical ork-rubric/1.0 + identical sealed set, with CLAUDE_CODE_FORK_SUBAGENT=1 so the only variable is the version under test. Both --bare constraints apply (requires ANTHROPIC_API_KEY, bills tokens directly → on-demand / CI only). Run it with bash tests/evals/scripts/run-skill-eval.sh --holdout-promote <skill>.
Related
eval:skillnpm script — unified skill evaluation runnereval:trigger— trigger accuracy testingeval:quality— A/B quality comparisonoptimize-description.sh— iterative description improvement- Version compatibility:
doctor/references/version-compatibility.md
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/yonatangross/orchestkit/bare-eval">View bare-eval on skillZs</a>