Every Model Explained Is Every Model Misleading: The Evaluation Problem

01 The Landscape Genre

Every few weeks a new explainer promises the same service: the current AI model landscape, explained. Which model writes best, which codes best, which is worth a subscription. The genre performs a real function — the model market is genuinely confusing, with overlapping names, version drift, and marketing claims that outrun documentation — and the better entries in the genre are careful, current, and honest about uncertainty.

The genre also has a structural flaw that no amount of care fixes: it ranks models as if ranking were a stable property of the models. In software, version numbers imply that a comparison holds until the next version. In frontier AI, the comparison decays within weeks — and the decay is not an accident of sloppy journalism. It is what evaluation on fast-moving systems mathematically does.

02 Why Rankings Feel True and Stop Being True

A benchmark is a fixed question set with a scoring rule. When a new model scores higher, the natural reading is that it is better. That reading was defensible when models changed slowly and benchmarks were young. It stops being defensible for two reasons that compound: the questions leak, and the models are trained on the leaks.

Contamination is the open secret of modern evaluation. Any public benchmark eventually becomes training data — its questions get posted, discussed, mirrored, and absorbed. A model that has memorized near-duplicates of a test set scores brilliantly without being able to do the thing the test was designed to measure. The score rises; the capability it was a proxy for does not.

03 Saturation: When a Test Stops Discriminating

The second mechanism is saturation. When every frontier model scores ninety-plus on a benchmark, the benchmark stops separating systems and starts flattering them. The remaining spread comes from ordering effects and item quirks rather than capability. Leaderboards respond by rotating in harder tests, but the replacements are born contaminated — they are public on day one, and frontier labs have every incentive to tune against them before shipping.

The result is a treadmill: benchmarks must be replaced faster than models improve, or they stop measuring. Private held-out sets help, but a held-out set cannot be audited by the community, and an unaudited eval is a marketing claim with a spreadsheet.

04 The Proxy Gap

Underneath both mechanisms sits the deeper problem: every benchmark measures a proxy. Exam scores proxy for reasoning, code-contest ratings proxy for software ability, preference win rates proxy for user satisfaction. Proxies are chosen because they are measurable, and they stop tracking the target the moment systems are optimized against the proxy rather than the target.

This is Goodhart's law operating at industrial scale and quarterly cadence. When a proxy becomes a target — and leaderboard position is nothing if not a target — the correlation that made the proxy useful is spent. The eval that launched a thousand launch posts is usually already spent by the time the posts are written.

05 What Careful Readers Do Instead

None of this means model comparison is impossible; it means comparison must be done against a task, not against a field. The informative question is never 'which model is best' but 'which model does this workflow best, on my data, at my price, with my latency budget.' That question has an answer, it is measurable, and it is stable for exactly as long as the workflow is.

The practical protocol is unglamorous: pick ten to fifty real examples from the actual workload, run the current candidate models on them, score with a rubric a person wrote, and re-run whenever a model version changes. This is what serious engineering teams already do, and it explains why their model choices often disagree with the leaderboard of the month.

06 Reading the 2026 Field Without Being Fooled

The honest way to consume a landscape explainer is as a map of what exists, not a ranking of what wins. Names, pricing tiers, context windows, and positioning are stable enough to communicate. Superlatives are not. The moment an explainer says 'best', it is publishing a snapshot of a benchmark that is already decaying.

For the models themselves, the field-level takeaway of 2026 is that the top tier is genuinely crowded: several labs ship systems within noise of each other on most real workflows, and the differences that matter are context handling, tool reliability, price, and distribution — not benchmark deltas. The evaluation problem is the market's problem now: as long as buyers demand rankings, the industry will supply them, calibrated or not.