LLM evals are repeatable tests for AI features: a set of inputs, a way to grade each output, and a score you track every time you change a prompt, model or tool. OpenAI's evaluation best practices put it in one line: "Evals are structured tests for measuring a model's performance."
Without evals, you judge prompt changes by trying a few examples and deciding it "seems better." OpenAI lists that as an anti-pattern, "Vibe-based evals." This guide covers how to define success, build a test set, choose graders, and evaluate agents, drawing on Anthropic's and OpenAI's own guidance.
Why do LLM apps need evals?
LLM outputs are not deterministic. The same prompt can produce different answers on different runs, and a change that fixes one case can break three others. Regular unit tests expect one exact output, so they cover only part of the problem.
Evals give you a number to compare. Typical moments to run them:
- You edit a system prompt or add an example
- A new model version comes out and you want to know whether to switch
- You add a tool, a retrieval step or a subagent
- A user reports a bad answer and you want to stop it from coming back
OpenAI's advice is to "Adopt eval-driven development: Evaluate early and often." Anthropic's engineering post on agent evals (9 January 2026) makes the same case: good evals let teams ship agents with more confidence instead of finding problems only in production.
What are the parts of an eval?
Anthropic's agent evals post defines the vocabulary most teams now use:
| Term | Anthropic's definition |
|---|---|
| Task | "a single test with defined inputs and success criteria" |
| Trial | "Each attempt at a task is a trial." |
| Grader | "logic that scores some aspect of the agent's performance" |
| Transcript | "the complete record of a trial, including outputs, tool calls, reasoning" |
| Outcome | "the final state in the environment at the end of the trial" |
| Evaluation harness | "the infrastructure that runs evals end-to-end" |
| Eval suite | "a collection of tasks designed to measure specific capabilities or behaviors" |
For a simple prompt, a task is one input and an expected answer. For an agent, a task might be "fix this failing test," and the outcome is whether the test suite passes afterward.
How do you build an eval step by step?
OpenAI's process has five steps: "Define eval objective," "Collect dataset," "Define eval metrics," "Run and compare evals," and "Continuously evaluate." In practice:
1. Define success. Anthropic's guide to success criteria and evals says criteria should be specific ("Instead of 'good performance,' specify 'accurate sentiment classification.'") and measurable, using numbers or "well-defined qualitative scales."
2. Collect cases. Start small and real. Anthropic's agent post says "20-50 simple tasks drawn from real failures is a great start." Its research agent team "started with a set of about 20 queries representing real usage patterns." Include edge cases, and make tasks unambiguous: two experts should agree on whether an output passes.
3. Write a reference answer. For each task, Anthropic recommends creating "a known working output that passes all graders." If you cannot write one, the task is probably unclear.
4. Choose graders. See the next section.
5. Run, compare, repeat. Run the suite on your current version to get a baseline, then rerun on every change. Add each new production failure as a case.
Anthropic's design principles for the set: "Be task-specific," "Automate when possible," and "Prioritize volume over quality." The last one means many automatically graded cases beat a handful of hand-graded ones.
How do you grade LLM outputs?
There are three kinds of graders. Anthropic's agent evals post sums up their tradeoffs:
| Grader | Strengths | Weaknesses | Good for |
|---|---|---|---|
| Code-based | "Fast," "Cheap," "Objective," "Reproducible" | "Brittle to valid variations," "Lacking in nuance" | Exact answers, JSON shape, tests passing |
| Model-based (LLM judge) | "Flexible," "Scalable," "Captures nuance" | "Non-deterministic," "More expensive than code" | Tone, completeness, faithfulness |
| Human | "Gold standard quality" | "Expensive," "Slow" | Calibrating other graders, high-stakes checks |
Anthropic's evals guide says to "choose the fastest, most reliable, most scalable method," and describes human grading as "Most flexible and high quality, but slow and expensive. Avoid if possible."
Code graders cover more than you might expect. The same guide lists exact match and string match, and its worked examples add cosine similarity for consistency and ROUGE-L for summaries. Its exact-match example is two lines:
def evaluate_exact_match(model_output, correct_answer):
return model_output.strip().lower() == correct_answer.lower()
If you need machine-checkable outputs, structured outputs make code grading much easier, because you can compare fields instead of parsing prose.
How do you use an LLM as a judge reliably?
An LLM judge reads an output and a rubric and returns a grade. Anthropic's tips:
- "Have detailed, clear rubrics." Spell out what passes and what fails.
- "Be empirical or specific." Ask for "correct" or "incorrect," or a 1 to 5 score, not an essay.
- "Encourage reasoning." Use a grader model with thinking on, which the guide says "increases evaluation performance, particularly for tasks requiring complex judgment."
A judge prompt following those tips might look like this:
You are grading a support reply.
Rubric:
- Pass only if the reply answers the customer's actual question.
- Fail if it invents a policy, a price or a feature.
- Fail if it is longer than 150 words.
Reply to grade:
<reply>
...
</reply>
Think through each rubric line, then output PASS or FAIL in <result> tags.
OpenAI's guide adds two cautions. Judges have known biases: "Position bias (response order), verbosity bias (preferring longer responses)." And you should "Use human feedback to calibrate automated scoring." Grade a sample by hand, compare with the judge, and fix the rubric where they disagree.
Anthropic's post is blunt about checking your graders: "You won't know if your graders are working well unless you read the transcripts and grades from many trials."
How do you evaluate agents?
Agents are harder to test because they take many steps, call tools and change state. Anthropic's agent evals post offers several rules.
Grade the outcome, not the path. "It's often better to grade what the agent produced, not the path it took." An agent that fixes the bug by a route you did not expect should still pass.
Run several trials. Because results vary, run each task more than once and report:
- pass@k: "the likelihood that an agent gets at least one correct solution in k attempts."
- pass^k: "the probability that all k trials succeed."
pass@k fits tasks where a human picks the best attempt. pass^k fits tasks where users need the agent to succeed every time.
Split capability and regression suites. Capability evals ask "What can this agent do well?". Regression evals check that the agent still handles what it used to, and should pass at close to 100%.
Multi-agent systems need this most, since errors compound across agents. See multi-agent systems for how Anthropic tested its research agent.
What tools can run LLM evals?
You can start with a script: a list of cases, a loop that calls the model, and a grader function. Anthropic's evals guide includes full code examples in several languages.
Hosted options change often, so check current docs before you commit:
- OpenAI: the Working with evals guide describes the Evals API, where
data_source_configis "A schema for the test data" andtesting_criteriaare "The graders that determine if the model output is correct." The same page now says "OpenAI is deprecating the Evals platform," with read-only access from October 31, 2026 and shutdown "scheduled" for November 30, 2026. It suggests trying Datasets instead. - Google agent tooling: the Google Agents CLI eval skill teaches coding agents to evaluate agents with Google's agents-cli: datasets, metrics and LLM-as-judge scoring.
- Open-source frameworks: many exist; pick one that stores cases as plain files so they live next to your code.
Whatever you use, keep the test cases in version control so a prompt change and its eval results can be reviewed together.
How do evals fit with your prompt library?
An eval only means something if you know which prompt version it tested. Keeping prompts as files, next to their test cases, makes that easy to track. Kipt stores prompts and Agent Skills as Markdown in a private GitHub repo you own, so the prompt you evaluated is a file you can find, compare and reuse.
For the prompt side of the loop, read the prompt engineering guide. For a first eval, pick one feature, write 20 cases from real inputs, add a code grader where you can and an LLM judge where you cannot, and run it before your next prompt change.