Using a language model to grade another model’s outputs against a rubric or reference answer—scalable and cheaper than human review, and the backbone of automated evals when the correct answer is open-ended.
LLM-as-judge uses a strong model to score outputs that have no single exact-match answer—summaries, agent trajectories, chat replies—against a rubric, a reference answer, or a pairwise "which is better?" comparison. It scales evaluation far past what human review can cover, which is what makes it practical to run a golden dataset on every change in CI.
Caveats
The judge is itself a fallible model. It carries known biases (position bias, length bias, self-preference for its own family's style), so a judge must be validated against human labels before you trust its scores—otherwise you are measuring the judge, not the system. Pin the judge model and version it alongside the rubric to keep scores comparable over time and reduce verification debt. Pairs naturally with LLM benchmarking and observability.