Running a model or agent through a fixed set of tasks with a scoring rule, so different options can be compared on the same yardstick instead of by gut feel—but a high leaderboard score only matters if the tasks look like your real work.
Benchmarking means measuring an AI system on a fixed set of tasks with a defined way to score the results. That lets you compare two models—or two versions of your own agent—on the same yardstick, rather than by vibes.
Different benchmarks measure different things:
- Knowledge and reasoning — broad question sets that probe what a model "knows."
- Coding — write-the-function tests such as HumanEval, where the code is run to check it actually works.
- Agentic / computer-use — whole tasks an agent has to complete end to end. OSWorld is the best-known example: it scores an agent on real computer tasks—opening apps, clicking, typing, working with files—so it measures how well a model can act, not just answer. This is the benchmark that jumped from roughly 12% success in early 2025 to about 66% in 2026.
Pitfall
A public leaderboard is a hint, not a verdict. A model that tops a benchmark built on someone else's code, stack, or languages can still fall flat on yours. Pair a small, fixed suite of your own tasks with blinded human ratings and production observability.