The Illusion of Thinking: Inside the LLM Reasoning Debate

01The paper that lit the fuse: what Apple's researchers actually tested

In June 2025, researchers then at Apple published The Illusion of Thinking, an experiment heavy critique of how we evaluate so-called reasoning models. Instead of static benchmarks, which leak into training data and saturate, they used controllable puzzle environments: the Tower of Hanoi, river-crossing problems and blocks-world jumping, each scalable in difficulty by adding pieces.

The design compared standard large language models with their reasoning-enabled counterparts across the same problems, tracking accuracy and, unusually, the models' internal chain-of-thought traces. The framing was deliberately provocative: if celebrated reasoning behaviour disappears at higher complexity, what exactly is the reasoning?

02Tower of Hanoi and the collapse point: what the accuracy curves showed

The Tower of Hanoi is a clean probe because its difficulty is exactly quantifiable: solving for n disks requires 2 to the n minus 1 moves, so nine disks demand 511 perfectly ordered steps. The reported pattern was striking. At low disk counts reasoning models held near-perfect accuracy, then accuracy collapsed sharply past a complexity threshold, and the models reduced their reasoning effort exactly where the problems got hardest.

Three regimes emerged in the reported curves. Standard and reasoning models performed similarly on simple tasks, reasoning models bought a margin at moderate complexity, and beyond the collapse point both families fell toward zero, with the reasoning models sometimes abandoning their chain-of-thought entirely. The flat-then-collapse shape is the image the debate has hung on ever since, and it is the pattern our first chart redraws illustratively.

03The rebuttals: token limits, estimation shortcuts and disputed setups

The pushback came fast and was technically specific. Several groups showed that part of the collapse was mechanical: output token budgets ran out on long Hanoi solutions, so the model was cut off rather than confused. Others found models switching to closed-form shortcuts, computing the number of moves rather than simulating them, which looks like failure under a step-by-step scorer while being a reasonable strategy.

The river-crossing tasks drew the sharpest criticism, since known instances are trivially easy for humans and the invented instances changed the rules in ways that reward ignoring real-world priors. The authors responded to methodological points, and in later revisions reframed some claims. The honest reading is that the headline collapse was real in the tested setups but over-interpreted as a general law.

04Chain-of-thought versus actual computation: why the distinction matters

The deepest question under the fight is whether a chain of thought is computation or commentary. If the visible reasoning steps genuinely mediate the answer, then longer problems should benefit from longer traces, and traces should be faithful to how the answer was reached. If the trace is a rationalised sketch and the answer arrives elsewhere, then reasoning scores measure something closer to retrieval.

The Hanoi evidence cuts both ways. Truncating the trace hurts, which suggests it does work; but effort collapsing precisely when problems get harder suggests the model's control policy misjudges when to invest. For deployers the distinction is not academic: whether chains of thought are load-bearing determines whether you can trust them for auditing, whether longer budgets help, and how far a system will generalise beyond its training distribution.

05What 'reasoning models' add - and what they still lack

Reasoning models trained to produce extended deliberation before answering clearly move the frontier on mathematics, code and planning-style problems. They buy their gains with latency and tokens, and the illusion-of-thinking results set a boundary on that purchase: deliberation helps within a complexity band and does not rescue a model past its compositional limits.

What they still lack is guaranteed systematicity. No current architecture promises that solving 6-disk Hanoi implies solving 9-disk Hanoi, because competence is learned statistically rather than implemented as an algorithm. Hybrid systems that call verified solvers as tools sidestep this entirely, which is why tool-augmented approaches have gained credibility from exactly this literature.

06The pattern-matching counterargument: humans memorize too

The strongest philosophical rebuttal inverts the charge. Humans also solve the Tower of Hanoi by learning a recursive procedure, and students memorise the parity rule rather than deriving it each time. If memorised strategy disqualifies a machine, it disqualifies the species, so the argument goes; the fair comparison is learning curves and error patterns, not a purity test.

The authors' reply is that the question is architectural: a system whose competence is bounded by the distribution of its training examples fails differently than one that acquires a generative rule. That distinction survives the rebuttal, but it moves the burden of proof to measurement: nobody currently has a test that cleanly separates learned procedure from memorised cases at frontier scale.

07Implications for AI safety and deployment decisions in 2026

The practical lesson for 2026 deployments is about calibration, not prohibition. Reasoning models should be assigned tasks whose complexity you can estimate, with complexity monitoring and fallbacks for the regime past the collapse, and with chain-of-thought treated as evidence rather than gospel in high-stakes review.

For safety work the paper reinforced an uncomfortable norm: capability claims built on saturated benchmarks hide the boundaries that matter. Puzzle-style probes, however artificial, made a boundary visible that standard evaluations had smoothed over. Expect regulatory and enterprise evaluations to keep a slice of controllable-complexity tasks for exactly that reason.

08Where the science stands: open questions and next experiments

The field has moved past the initial polemic toward narrower, better-controlled questions. Open items include separating token-budget effects from genuine reasoning failure, testing whether collapse points shift with scale or training method, and checking faithfulness between reported traces and computed answers.

What would settle the debate is less dramatic than a headline: pre-registered experiments with adequate budgets, solvers in the loop as ground truth, and comparisons across architectures. Until then the fair summary of the illusion-of-thinking affair is that it demonstrated a real, reproducible edge case, mislabeled it as a universal law, and in doing so taught the field how to test reasoning claims more honestly.