Evaluation & Benchmarks
How do we know if a model is any good? We test it — on benchmarks: fixed sets of math problems, coding tasks, exam questions. But a benchmark is one narrow lens, and every headline claiming ‘the best model’ is really claiming ‘the best score on one particular test’.
Switch between benchmarks and watch the same three models reshuffle: A tops math, B tops coding, C tops Hindi Q&A, and human preference barely separates them. Real complications go further: models may have SEEN benchmark questions during training (contamination), scores saturate as everyone optimises for the same tests, and human raters reward confidence and formatting over correctness.
A dominates — but was it trained on similar problems? Contamination is hard to rule out.
Every benchmark is one narrow lens. "State of the art" always means: on this test, on this day.
Check yourself
A new model beats all others on a famous benchmark. Give two reasons this might not mean it is better for YOUR task.
Go deeper (free): Hugging Face — LLM evaluation guidebook ↗