TL;DR
An eval dataset is a named set of cases — each an input, an optional expected answer, and the metric(s) that decide pass/fail. The best cases come from real failures: capture a question your agent got wrong, add it as a case, and that exact bug can never ship unnoticed again.
A dataset is just curated cases
An eval dataset is a named set of cases. Each case is an input, an optional expected answer, and the metric(s) that decide pass or fail (last two lessons). Run the dataset against your agent and you get a score — and, case by case, exactly what passed and what didn't.
A good dataset is small enough to run often and broad enough to be honest. The seeded "Core suite" you'll use in the lab is twenty customer-support cases — a mix of happy-path questions and a few deliberately tricky ones.
Where good cases come from
You don't invent a great dataset at a desk. The best cases come from real failures — a question a user actually asked that your agent got wrong. You saw it, so you capture it: write the input, set the expected answer, choose a metric, and add it to the suite. Now that exact bug can never ship again unnoticed.
That loop — notice a failure → add it as a case → it's guarded forever — is how a dataset earns its keep. Every production surprise becomes a permanent test. That's the checkpoint: a real user failure is a case you know matters and know your agent got wrong, while a desk-invented case is a guess about what might matter — and it's usually a guess from the same blind spot that let the bug through.
Analogy
A dataset grown from real failures is like a hospital's collection of past misdiagnoses — each one a case the staff know is hard because they got it wrong once. A list of textbook symptoms you imagined is far weaker: it tests what you already expect, not what actually trips you up.
What makes a case good
- Specific input. The real phrasing, not a cleaned-up version.
- An honest expected answer (or a rubric, for judge cases) — what should have happened.
- The right metric for the answer shape (Module 7 lessons 2–3).
- A reason it's here. Happy-path coverage for confidence; tricky cases for the long tail. Know which each one is.
Common mistake
Building a dataset only from happy-path questions because they're easy to write. A suite that's all softballs gives you a comforting green score and catches nothing — it's the demo trap (Lesson 1) with extra steps. Deliberately seed the hard cases: the weird phrasings, the out-of-scope asks, the ones that already burned you.
Key takeaways
Key takeaways
- A dataset is curated cases: input + expected + metric, run for a score.
- Keep it small enough to run often, broad enough to be honest.
- The best cases come from real failures — capture them so they're guarded forever.
- Mix happy-path (confidence) with deliberately tricky cases (long-tail coverage).
Go deeper: keeping a dataset honest over time
A dataset isn't write-once — it needs tending, or it quietly stops measuring what you care about:
- Beware overfitting to the suite. If you tune the prompt until the suite is all green, you've optimised for those twenty cases, not for quality. Keep adding fresh real-world failures so the suite stays ahead of your tuning.
- Hold out a slice. Some teams keep a portion of cases the agent's authors never see while iterating, used only for a final check — the eval equivalent of a test set vs. a training set, to catch overfitting.
- Curate, don't just accumulate. Prune cases that no longer reflect real usage, and de-duplicate near-identical ones, so the score stays meaningful and the suite stays fast.
- Label the hard ones. Tagging cases (happy-path, edge, safety) lets you read the score by category — "we're 100% on happy-path but 60% on edge cases" is far more actionable than one blended number.
A dataset is a living asset. The next lesson is what you do with it: run it, and compare runs to catch regressions.