TL;DR
LLM-as-judge hands an answer to a model with a rubric and asks for a PASS/FAIL (or a score). It's the only metric that can grade judgement calls — tone, completeness, safety, faithfulness — but it's only as good as the rubric you write, and because the judge is itself an LLM, a vague rubric makes it inconsistent.
When a string check isn't enough
The deterministic metrics (last lesson) check form: exact string, keywords, embedding distance. Plenty of real quality questions aren't about form:
- Did the answer use a polite, on-brand tone?
- Was it complete — did it cover every part the user asked?
- Did it refuse an unsafe or out-of-scope request?
- Is it faithful to the retrieved sources, or did it add unsupported claims?
None of these is a substring or a cosine score. They're judgements — so you ask a model to make them, the way you'd ask a careful human grader.
The rubric is the metric
An LLM judge is only as good as the rubric you give it. A vague rubric produces a vague, swingy grader; a sharp rubric produces a consistent one. Compare:
| Rubric | What the judge does |
|---|---|
| "Is this a good answer?" | Inconsistent — 'good' is undefined, so the same answer can pass or fail run to run |
| "PASS only if the answer (1) states the refund window in days and (2) names the Billing tab. Otherwise FAIL." | Consistent — concrete, checkable criteria the judge applies the same way every time |
That's the checkpoint: a judge that flips on the same answer is telling you the rubric is underspecified. The fix isn't a "better model" — it's a rubric with concrete, enumerated criteria, so there's nothing left to interpret.
Analogy
Handing a judge a vague rubric is like telling a contest judge "pick the best one" with no criteria — you'll get a different winner each time, swayed by mood. Give them a scorecard ("10 points for X, 5 for Y, disqualify if Z") and any judge reaches the same verdict. The scorecard is the metric.
Writing a good rubric
- Enumerate concrete criteria, not adjectives. "States the policy and cites a source" beats "is helpful."
- Make it binary where you can. PASS/FAIL with explicit conditions is more stable than a 1–10 score, where the judge has to invent the scale.
- Ask for a reason. Having the judge explain why before the verdict improves consistency and gives you something to read when it disagrees with you.
- Show an example of a passing and a failing answer if the criteria are subtle.
Common mistake
Trusting the judge blindly because it's "an AI grading an AI." The judge has biases — it can favour longer answers, reward confident phrasing, or drift if the rubric is loose. Spot-check the judge against your own verdicts on a handful of cases before you rely on it across the whole suite. Calibrate the grader, then scale it.
Key takeaways
Key takeaways
- LLM-as-judge grades judgement calls (tone, completeness, safety, faithfulness).
- The rubric is the metric — vague rubric, inconsistent judge.
- Prefer concrete, enumerated, binary criteria; ask the judge to justify its verdict.
- The judge has biases — calibrate it against your own grading before trusting it at scale.
Go deeper: judge cost, determinism, and pairwise grading
A few realities once you run a judge across a real suite:
- Cost and latency are per case. Every judged case is a model call on every suite run. Use the judge where judgement is genuinely needed and deterministic metrics everywhere else, or your suite gets slow and expensive to run often.
- Determinism is a dial. A judge call at low temperature is more repeatable; some teams run the judge a few times and take the majority vote for high-stakes cases. Pin the judge model and prompt so a "regression" is a real change, not judge drift.
- Absolute vs. pairwise. Asking "does this answer PASS the rubric?" (absolute) is the common form. For comparing two systems, pairwise judging ("is answer A or B better?") is often more reliable than scoring each alone, because relative judgements are easier than absolute ones — useful when you're choosing between two prompts or models.
The throughline: an LLM judge is a powerful, flexible metric, but it's a fuzzy instrument grading a fuzzy system. Treat its rubric like code and calibrate it like a measurement device.