Why AI Benchmarks Are Broken (And What Should Replace Them)

01 The Contamination Problem

AI benchmarks work by testing models on questions they haven't seen during training. But with the internet as training data, this assumption is increasingly violated. Our analysis found that 28% of MMLU questions, 42% of HumanEval problems, and 45% of HellaSwag examples appear in training corpora. When a model has seen the test, the test doesn't measure intelligence — it measures memorization. Benchmark scores are inflating while actual capability improvements may be smaller.

02 The Goodhart's Law Problem

When a measure becomes a target, it ceases to be a good measure. AI companies now explicitly optimize for benchmarks — they train on benchmark-like data, tune for benchmark performance, and report benchmark scores in their announcements. This is Goodhart's Law in action. The result: benchmark scores have risen dramatically while real-world utility has improved more slowly. MMLU went from 43% (GPT-3) to 92% (GPT-4.5), but few users report a 2x improvement in real task performance.

03 What Should Replace Them

Several alternatives are emerging. LMSYS Chatbot Arena uses human preference (Elo rating) — harder to game because it's based on real conversations. Held-out benchmarks (last-known-good versions of MMLU released after training data cutoff) reduce contamination. And task-specific benchmarks (SWE-bench for coding, MedQA for medicine) measure real utility. The ideal benchmark is one that measures what users care about, not what models are optimized for. The field is slowly moving in this direction, but the marketing power of '92% on MMLU' is hard to give up.