How to test Claude skills: trigger tests, skill-creator evals and claude plugin eval

Why test a skill at all?

To test a Claude skill, check two things separately: whether Claude invokes it on the prompts it should, and whether the result is better than what Claude produces without it. You can do the first by hand in a minute; for the second, use skill-creator's evals or the claude plugin eval command.

Anthropic's Claude Code skills docs put it clearly: "Seeing a skill trigger tells you Claude found it, not that it did what you intended." A skill can fire every time and still make the output worse, or do great work on the rare occasions it fires.

Both checks use the same idea, a baseline comparison. Run realistic prompts in fresh sessions with the skill available, run them again with it off, and compare. Fresh sessions matter because "leftover context from authoring the skill will mask gaps in the written instructions."

There are three levels of effort, and most skills only need the first two:

How do you check that a skill triggers?

Start with a quick manual check in Claude Code:

1. Run /skills or ask "What skills are available?" to confirm the skill loaded.
2. Open a fresh session and type a request that matches the description in your own words, without naming the skill.
3. Watch whether Claude invokes the skill. If not, run /skill-name directly to confirm the instructions work at all.
4. If /skill-name works but Claude never picks the skill, the description is the problem. A broken YAML header also causes this: Claude Code loads the body with empty metadata. Run with --debug to see the parse error, or run claude plugin validate .claude/skills (v2.1.233 or later) to find headers that don't parse.

One prompt tells you little, because models are not deterministic. The Agent Skills project's guide to optimizing descriptions recommends a small labeled set instead: "Aim for about 20 queries: 8-10 that should trigger and 8-10 that shouldn't." Run each one three times and compute a trigger rate. With 0.5 as the threshold, a should-trigger query passes above it and a should-not-trigger query passes below it.

[
  { "query": "turn the PRs merged since v2.3 into notes for the changelog", "should_trigger": true },
  { "query": "what shipped this sprint? write it up for customers", "should_trigger": true },
  { "query": "summarize the review comments on PR 412", "should_trigger": false },
  { "query": "write a commit message for my staged changes", "should_trigger": false }
]

The best negative cases are near misses: prompts that share words with your skill but need something else. The same guide suggests holding back about 40% of the queries as a validation set, so you don't tune the description to fit only the phrasings you tested.

How do skill-creator evals work?

The skill-creator plugin automates the with and without comparison inside a Claude Code conversation. Install it from the official marketplace:

/plugin install skill-creator@claude-plugins-official

Then ask Claude something like "evaluate my release-notes skill with skill-creator." Per the Claude Code docs, the plugin then:

• stores prompts, input files and expected behavior in evals/evals.json inside the skill folder
• runs each test case in its own subagent so it starts with a clean context, recording tokens and duration
• grades each assertion with evidence in grading.json
• writes pass rate, time and tokens for with skill versus without skill to benchmark.json
• can run a blind A/B comparison between two versions of the skill
• generates should-trigger and should-not-trigger prompts and proposes description edits

The test case format comes from the Agent Skills guide to evaluating skill output. Start with two or three cases and add assertions after you've seen real output:

{
  "skill_name": "release-notes",
  "evals": [
    {
      "id": 1,
      "prompt": "Write release notes for everything merged since the v2.3.0 tag",
      "expected_output": "Notes grouped as New, Improved and Fixed, one user-facing line per change",
      "assertions": [
        "The output has New, Improved and Fixed headings",
        "Dependency bumps are not listed",
        "Each line starts with a verb"
      ]
    }
  ]
}

The benchmark's delta shows what the skill costs and what it buys. A skill that adds a few seconds but lifts the pass rate a lot is worth keeping; one that doubles tokens for a tiny gain may not be. Our skill-creator guide covers the rest of its workflow.

What is claude plugin eval, and how do you set it up?

claude plugin eval is a shell command for repeatable regression tests. Anthropic's Test plugins with evals page lists the requirements: Claude Code v2.1.269 or later, Git 2.31 or later if Git is installed, and a plugin directory or a skills-directory plugin. A skills-directory plugin is a skill folder under .claude/skills/ or ~/.claude/skills/ that also has a .claude-plugin/plugin.json.

1. From the plugin root, run claude plugin eval init. Claude asks what a good result looks like, proposes prompts that should and shouldn't trigger the plugin, designs graders, tries them once and writes one case folder per prompt under evals/.
2. Run the suite with claude plugin eval ..
3. Read the summary table and open the report it links.

Each case is a folder with a prompt.md and one file per grader:

---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
---

Write release notes for these merged PRs: #401 add dark mode, #402 fix login loop, #405 bump eslint.

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?release-notes"'
---

Each case runs three times by default, because one run of a nondeterministic agent tells you little. A run's score is the share of graders that passed, and a case passes when its mean score meets --threshold, which defaults to 1.0. Runs are isolated: your settings, CLAUDE.md, MCP servers and other skills don't load, and the agent can't read the case files.

Which graders should you use?

There are six grader types. Four are computed from the transcript and files and cost nothing; two call a judge model:

The docs give a simple pattern: one grader on the result, such as the final message or a produced file, and one on the steps, such as tool_used for your skill. Use regex for long generated files and keep llm rubrics for short outputs, written as concrete PASS and FAIL conditions. There are no custom-code graders.

How do you compare against a no-skill baseline?

claude plugin eval repeats every case with no plugin loaded and reports WITH, W/OUT and their difference, Δ. If a case scores 1.0 both ways, the plugin isn't what made it pass.

Some graders can never pass without the plugin, such as tool_used with tool: Skill. To keep the two arms fair, those are excluded from the score in both arms and shown as a "plugin-fired indicator" only. The most common first finding, per Anthropic, is a Δ near zero with that indicator failing, which means Claude isn't choosing your skill on natural phrasing. Fix the description and run again.

For skills outside a plugin, make the baseline yourself: set the skill to "off" under skillOverrides in settings, or use skill-creator. To save money while you iterate on graders, --ablation none runs only the with-plugin arm.

How do you run skill evals in CI?

Run the suite with JSON output, pin both models so scores are comparable over time, and fail the build on the exit code. This is Anthropic's documented pattern:

claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20

Every run and judge call is a real model call on your account. Keep quick per-change suites to free graders, and save judge graders for a nightly run.

Codex users can follow the same ideas. OpenAI's post on testing Agent Skills with evals suggests 10 to 20 prompts in a CSV with a should_trigger column, deterministic checks on the JSONL events from codex exec --json, and a rubric pass using --output-schema.

After a skill ships, keep watching it. In Claude Code, /skill-doctor (v2.1.252 or later) shows each skill's context cost and flags ones that have never been invoked. Testing also proves behavior, not safety, so read Agent Skills security before you trust someone else's skill. For eval design in general, see our LLM evals guide.

Keep tested skills versioned

Evals are most useful when you can tie a score to an exact version of the skill. Kipt keeps your prompts and skills as Markdown in a private GitHub repo you own, so every change to a SKILL.md is a commit you can test against, and the whole library installs into Claude Code or Codex with npx skills add https://github.com/<you>/<repo>.git --all -g. To draft a new skill from a prompt that already works, start with the free SKILL.md generator, then follow how to write a SKILL.md.