TL;DR
Trying three questions in the playground only tests the happy path you already handle. An eval is an automated test for your agent — a dataset of inputs, each with a notion of a good answer, run against the real agent and scored. The catch: answers are free-form text, so you need a metric to decide pass/fail. Four metrics take you from strictest to smartest — and the metric you pick is your definition of "correct."
"It worked when I tried it"
You built an agent, asked it three questions in the playground, it answered well, and you shipped it. That's how almost every agent starts — and exactly how quality problems hide. Three demo questions are the happy path: the cases you thought of, which are the cases your agent already handles. The trouble lives in the long tail — the phrasing you didn't anticipate, the edge case you never tried, the question just outside what your prompt covers.
What an eval actually is
An eval runs a whole dataset through your agent and scores each answer, instead of you eyeballing one or two:
An eval runs every case in a dataset through the agent, scores each output with a metric, and aggregates a pass/fail result — which you re-run and compare after every change.
That loop — dataset → run → metric → score → compare — is the whole module. Each box is a lesson: scoring an answer (this lesson), the dataset (where cases come from), and the comparison (regression testing).
The demo trap
Here's an agent that looked finished on its demo questions. Switch it to the full suite and watch what happens:
Same agent. The only change is that we stopped grading it on the questions it was built for and started grading it on a representative set. Eight failures were there the whole time — you just couldn't see them.
Why this isn't a unit test
An eval sounds like a unit test, and the goal is the same — catch regressions, prove behaviour. But two things differ, and they're why scoring is the hard part:
- The output isn't deterministic. A unit test asserts
add(2,2) === 4. An agent asked the same question twice can word its answer differently both times and be correct both times. So evals can't just demand an exact match — they need fuzzier ways to score. - "Correct" is often a judgement. Was the tone right? Did it cite a source? Did it refuse the unsafe request? Some of that you check with a keyword; some you hand to another model to judge.
That's exactly what a metric is for.
How do you score a fuzzy answer?
A test needs a way to decide pass or fail. For an agent whose answers are free-form text, that decision is the whole game — pick the wrong metric and a correct answer reads as a failure, or a wrong one slips through. Four metrics, strictest to smartest:
| Metric | How it decides | Best for | Blind to |
|---|---|---|---|
| exact | Output matches the expected string char-for-char (trimmed, case-folded) | Closed answers: yes/no, a status code, one label | Any paraphrase — brutal on prose |
| contains | Output contains certain keywords | Cheap sanity check: "did it mention refund + 30 days?" | Whether the words were used correctly |
| semantic | Embed expected + actual, compare by closeness in meaning (Module 3) | Any faithful paraphrase should pass | Intent and correctness of specifics |
| llm_judge | A model grades the answer against a rubric → PASS/FAIL | Tone, completeness, safety — judgement calls | Only as good as the rubric; costs a model call |
Watch them disagree
The real lesson is that the metrics don't agree. Here's one input/output pair scored all four ways:
The answer is correct, but exact fails it on a wording difference, contains
passes because the keywords happen to be present, semantic passes because the
meaning matches, and llm_judge passes because its rubric asks the right question.
That disagreement is the checkpoint: a correctly-worded-differently answer fails
exact, squeaks past contains (if the keywords are there), and passes semantic
and llm_judge.
Analogy
The metrics are four graders with different standards. exact is a Scantron machine —
one bubble, no partial credit. contains is a TA skimming for keywords. semantic
is a reader who rewards "same meaning, different words." llm_judge is the professor
reading against a rubric. You pick the grader that matches what the question is
really testing.
Choosing a metric
The metric encodes what you actually care about:
- exact — when there's one right string and you mean it.
- contains — a cheap guardrail ("did it at least mention the policy?").
- semantic — when any faithful paraphrase should pass.
- llm_judge — when "good" is a judgement (tone, helpfulness, did-it-refuse). It's powerful enough to deserve its own lesson (next).
Most real suites mix them — a keyword check here, a judge there, a semantic comparison where paraphrase is fine. A single case can even require several to pass at once.
Common mistake
Defaulting to exact because it feels rigorous. On free-form answers, exact fails
every correct paraphrase — you'll drown in false failures and stop trusting the
suite. Match the metric to the answer shape: exact only for closed/single-string
answers, fuzzier metrics for prose.
Key takeaways
Key takeaways
- Demo questions test the happy path; failures hide in the long tail — an eval scores a whole representative dataset, not one or two tries.
- Agent output is non-deterministic and "correct" is often a judgement, so you score with a metric, not an exact match.
- exact → closed answers; contains → keyword guardrail; semantic → paraphrase-tolerant; llm_judge → judgement calls.
- Metrics disagree on the same answer — choosing well is the skill; real suites mix them.
Optional — deterministic vs. model-graded metrics
The four metrics split into two families with very different properties:
- Deterministic (
exact,contains,semantic): same inputs always give the same score, run locally, cost ~nothing, and are fast. Their limit is that they measure form (string, keywords, meaning-closeness), not judgement. - Model-graded (
llm_judge): can assess things no string check can — tone, safety, completeness — but it's a model call (latency + cost), and it can be non-deterministic itself (the judge is an LLM). You're using one fuzzy system to grade another.
The practical rule: reach for a deterministic metric whenever the property is
checkable by string/meaning, and escalate to llm_judge only for genuine judgement
calls. Every judge case you add is a model call on every suite run — cheap per case,
but it adds up across a large dataset run frequently. The next lesson is all about
using the judge well.
Optional — evals and traces are a flywheel
Evals don't stand alone — they close the loop you started in the Traces module:
- Traces tell you that one run failed and why (Module 6's symptom → span).
- Evals tell you how often it fails across many cases, and whether a change helped or hurt the whole suite, not just the one run you were staring at.
- Together they're a flywheel: a production failure shows up in a trace → you turn it into an eval case → the suite guards it forever → you change the prompt or model and re-run the suite to confirm you fixed it without breaking anything else.
This is why "ship and watch" isn't enough on its own: watching surfaces individual failures, but only a scored suite tells you whether your overall quality is going up or down as you iterate.