TL;DR
You run a dataset, get a score, change something, and re-run — then compare the two runs case by case. The cases that flip fail → pass are your wins; the ones that flip pass → fail are regressions. That comparison is what lets you change an agent with confidence instead of crossing your fingers.
A score is only useful next to another score
Running the suite once gives you a number — "16 of 20 pass." On its own that number is hard to act on. Its power shows up on the second run, after you change something: now you can compare. Evals turn "I think this prompt is better" into "this prompt flipped two failures to passes and introduced zero regressions."
The two flips that matter
When you re-run after a change, every case lands in one of four buckets — and two of them are the whole point:
| Before → After | Meaning |
|---|---|
| fail → pass | Your change fixed this case — a win |
| pass → fail | A regression — your change broke something that worked |
| pass → pass | Unaffected (still good) |
| fail → fail | Still broken (didn't address it) |
That's the checkpoint: two fail → pass and one pass → fail means your fix worked
but introduced a regression — and the regression is what you care about most,
because a silent pass → fail is exactly the breakage that ships when no one's
comparing runs. You'd investigate the regression before celebrating the wins.
Analogy
Re-running the suite is a before/after photo. The number alone ("16/20") is one photo — mildly informative. Side by side with the previous photo, the differences jump out: this got better, that got worse. Regression testing is looking at the diff, not the snapshot.
The regression safety net
This is the discipline the whole module builds toward. Without a suite, you change a
prompt to fix one problem and find out from users that you broke three others. With
a suite, you re-run and the pass → fail flips surface the breakage before it ships.
The suite is what makes iteration safe: you can refactor a prompt, swap a model, or
adjust retrieval and know whether you came out ahead.
In the lab you'll do exactly this: run the Core suite, find three failing cases, inspect each one's real trace to diagnose why it failed (Module 6), fix one by editing the system prompt, and re-run to confirm the fix landed without regressing anything else.
Common mistake
Judging a change by the headline score alone. "16/20 → 17/20, nice" can hide a
pass → fail regression masked by two new passes — the total went up while you
quietly broke a case that used to work. Always read the per-case diff, not just
the aggregate. The number going up is not the same as nothing breaking.
Key takeaways
Key takeaways
- A score is actionable mainly in comparison to a previous run.
- fail → pass = wins; pass → fail = regressions — watch the regressions hardest.
- The suite makes iteration safe: change boldly, re-run, read the diff.
- Always inspect the per-case diff, not just the aggregate score.
Go deeper: evals in CI and the non-determinism wrinkle
Running a suite by hand is the start; teams wire it deeper:
- Evals in CI. Run the suite automatically on every prompt/model change, the way
unit tests run on every commit. A drop below a threshold (or any new
pass → fail) blocks the change. This turns "remember to run evals" into "you can't merge without them." - The non-determinism wrinkle. Because agent output isn't deterministic, a case
can flip
pass → failpurely from run-to-run variance, not a real regression. Guard against false alarms by pinning the model and temperature, and for flaky cases, running them a few times and requiring a majority — so a single unlucky sample doesn't fail the build. - Thresholds, not perfection. Suites rarely hit 100% and shouldn't have to. Teams set a target ("≥ 90% pass, zero safety-case failures") and gate on that, plus the no-new-regressions rule — which matters more than the absolute number.
The endpoint: quality becomes a number you watch over time, with a safety net that catches regressions automatically. That's the whole reason the module exists — replacing "it worked when I tried it" with a measurement you can trust.