Writing Evals for a Skill

What it takes off your plate: killing "I think this version is better"

The first thing you can actually run

# Path A: the built-in runner. Scaffold a case skeleton from a plugin/skills root
claude plugin eval init --bare my-first-case
# -> writes evals/my-first-case/prompt.md and evals/my-first-case/graders/criteria.md

# Fill them in, then run the current directory three times with a self-contained report
claude plugin eval . --runs 3 --report ./eval-report.html

# Path B: install the official skill-creator and let it drive the whole loop
/plugin marketplace add anthropics/claude-plugins-official
/plugin install skill-creator@claude-plugins-official
/reload-plugins
# then just say: evaluate my summarize-changes skill with skill-creator

# Path C: the fallback. Turn the skill off in .claude/settings.local.json, rerun in a fresh session
{ "skillOverrides": { "summarize-changes": "off" } }

# Five-second checkup before any of this: did the frontmatter parse at all?
claude plugin validate .claude/skills

What init --bare emits is smaller than you would guess: prompt.md carries only max_turns and allowed_tools in its frontmatter, graders/criteria.md only type and weight, with a one-line TODO under each. A case is a directory, not a config file: one folder per case, holding a prompt and a graders/ directory.

The mechanism: how a feeling becomes a number

1. The baseline arm is on by default, not opt-in

--ablation defaults to with-without: whenever a plugin resolves, the runner adds an arm with no plugin at all and reports the score delta. Why must this be the default? Because the model is already strong. Your skill says "write commit messages in conventional-commits form" — the model may well do that with no skill loaded. Without the second arm you would book that point to yourself. skill-creator states the same rule even more bluntly: the with-skill and baseline runs must be spawned "in the same turn". No running the with-skill batch today and backfilling baselines next week.

2. Firing is not quality: with-only graders stay out of the score

This is the sharpest piece of the design. Under with-without, graders marked with-only — including tool_used: Skill — count as an indicator that the plugin fired, not as part of the score. The reasoning is plain: the arm without the skill can never fire the skill, so folding that into the delta is a free point you award yourself. Trigger rate and output quality must be measured separately, which is exactly why skill-creator has a distinct description-tuning step: it generates should-trigger and should-not-trigger prompts, measures the hit rate, and proposes description edits.

3. Two classes of grader: exhaust the free ones first

Put an invalid type in a grader and the CLI lists every legal value: regex | tool_order | tool_used | file_exists | llm | baseline. The first four are pure program checks — free, and they add no variance. The last two are model calls (--judge-model defaults to haiku). That line decides how you design cases: if success can be stated as "produced output.txt", "read before it edited", or "the output contains feat(", do not restate it as a vague "the response should be professional and clear" for an LLM to interpret. It is also precisely the official evaluation principle — more questions with slightly weaker automated grading beats fewer questions graded by hand.

4. The repeat count is 3 by default

--runs defaults to case.runs ?? 3. A single winning run tells you almost nothing in agent evaluation, because the same prompt can produce wholly different trajectories twice in a row. skill-creator carries this into the report: python -m scripts.aggregate_benchmark produces benchmark.json and benchmark.md giving pass_rate, time and tokens per configuration as mean +/- stddev, plus the delta. Read the stddev before the delta — a difference drowned in the standard deviation is not an improvement.

Real usage and boundaries: five steps, three cases, two Claudes

The five steps (source: Skill authoring best practices · Evaluation and iteration)

1. Identify gaps: run real tasks with no skill and write down exactly where it failed and what context was missing.
2. Create evaluations: build three scenarios around those gaps. skill-creator words it as "2-3 realistic test prompts — the kind of thing a real user would actually say", and explicitly says to write only the prompts first and draft the assertions later, while the runs are in flight — because you do not yet know how it will go wrong.
3. Establish baseline: measure performance without the skill.
4. Write minimal instructions: just enough to close the gaps and pass the evaluations, then stop.
5. Iterate: run the evaluations, compare against baseline, refine.

Claude A / Claude B, and why the session must be fresh

The official method is two instances: Claude A helps you write and revise the skill, Claude B has it loaded and does real work, and you carry B's failures back to A. The docs give a concrete case — "I asked B for a regional sales report; it wrote the query but forgot to filter out test accounts, even though the skill mentions that rule" — and A might respond by changing "always" to "MUST", or by reordering sections.

"Fresh session" is a hard requirement, not a flourish: the docs say plainly that context left over from authoring the skill will mask gaps in the written instructions. You think it understood the file; it actually remembers what you said out loud. Both claude plugin eval and skill-creator run cases in subagents for this reason — every case gets a clean context.

Three tools, three different measurements — do not confuse them

• claude plugin validate (v2.1.233+) — checks only that the frontmatter YAML parses. It catches a genuinely sneaky failure: the YAML breaks, the skill body still loads, but the metadata is empty, so /skill-name still works by hand while automatic triggering never happens again. Run it before any eval. Five seconds.
• /skill-doctor (v2.1.252+) — measures usage and context cost, not quality. It lists what each skill costs in context and how often it was invoked, and flags the ones never invoked at all. In an interactive session the report opens in the /plugin manager's Stats tab; in non-interactive mode with -p it prints as text. Where it will not run: over a Remote Control connection it answers "Skill usage reports are not available on this connection." Run it in the terminal on the machine hosting the session.
• skillOverrides — four visibility states (on / name-only / user-invocable-only / off) in .claude/settings.local.json; the /skills menu writes it for you (Space cycles, Esc saves). Use it as a poor man's ablation switch. Boundary: it has no effect on plugin skills — those go through /plugin.

The cost, and the alternatives: evals cost more than skills, and they rot

The honest costs

• The bill multiplies. Per case: runs x two arms x one full agent run each — 3 x 2 = 6 by default. The llm and baseline graders each add a model call on top. claude plugin eval offers --max-cost-usd as a hard ceiling; on a breach it aborts and reports partial results (exit 2), with the overrun bounded to a single agent run — and for that run the paid graders are skipped while the free ones still score it. Use it on your first run.
• Too tight and too loose both lie to you. Tight assertions give high variance and permanent red; loose ones give identical arms and a delta pinned to zero. Both present as "the evals are useless" while having opposite causes, which is what the analyst pass is for.
• Early-access tooling does not belong in CI yet. --threshold (default 1.0, exit 1 below it) and --json are obviously built for pipelines — but the command exits immediately on an account without access. Confirm it runs under the CI identity before you wire a gate to it.
• The docs contradict each other. The best-practices page still says "there is not currently a built-in way to run these evaluations", while the Claude Code docs carry a full evaluate-and-iterate section. When two official pages disagree, believe the command that runs.
• Evals rot. After a model generation turns over, cases that used to discriminate may pass without the skill too, and your delta shrinks to nothing on its own. The official requirement to test on Haiku, Sonnet and Opus is, in effect, a way to surface that early.
• No subagents means no baseline. skill-creator says so itself: in environments without subagents, skip the quantitative benchmark, because baseline comparison is not meaningful there — fall back to qualitative human feedback. Do not pretend to be evaluating in that setting.

What to use instead

• promptfoo — a general prompt / agent / RAG evaluator; per its GitHub repo page, 2026-09, roughly 24.9k stars. Install with npm install -g promptfoo, start with promptfoo init --example getting-started; it supports LLM-as-judge assertions and is built around CI integration. When not to use it: it knows nothing about Claude Code's skill triggering. It can score "is this output good", never "should this skill have fired".
• Inspect AI — the evaluation framework from the UK AI Security Institute; per its GitHub repo page, 2026-09, roughly 2.7k stars, and per PyPI version 0.3.263 released 2026-09-04. Install with pip install inspect-ai; it ships model-graded scorers and over 200 prebuilt evaluations. When not to use it: it is research-grade infrastructure for comparing model capability. Pointing it at one SKILL.md is artillery against a mosquito.
• The manual two-run — the fallback the official docs list themselves: a few realistic prompts, each run in a fresh session once with the skill and once with it turned off via skillOverrides, compared by eye. When that is enough: the skill is yours alone, the change was pure wording, or you only want to know whether it fires.