Thinking Tokens: The Economics of Paying AI to Reason Longer
Photo: N43 and Hermes AIThe frontier moved from training bigger models to letting them think longer at query time. That shift changes what a token costs, how fast an answer arrives, and which workloads benefit.
Source video: Building OpenAI o1 (Extended Cut) · OpenAI · approximately 281,628 views observed via yt-dlp on 2026-11-20. Independently researched by N43 and Hermes AI.
01 From bigger models to longer answers: the test-time shift
For most of the modern AI boom, progress had one axis: scale the training run. More parameters, more data, more GPU-months — capability followed compute spent before the model ever met a user. Reasoning models broke that assumption. Instead of moving the compute earlier, they move it later: at query time, the model generates intermediate tokens that are never shown to the user, enumerating steps, checking them, and revising before it commits to an answer. The training run gets smaller relative to the lifetime compute budget, because every hard question now carries its own private training session.
The industry shorthand for this is test-time compute, and it is less a single technique than a reallocation of the same budget. A reasoning language model is still a large language model — the architecture is the transformer every deployer already serves — but it is trained to spend its answer window productively: planning, trying an approach, noticing it fails, backing up. What looks from the outside like a pause before the reply is, mechanically, a stream of billed tokens the product has decided you should not see.
02 How a thinking token is made: RL on chains of thought
The mechanism that makes hidden tokens useful is training method, not architecture. Standard LLMs are trained to imitate text; reasoning models are trained with reinforcement learning against verifiable outcomes — a math answer is right or wrong, a unit test passes or fails. When a chain of intermediate steps ends in a verifiable success, the policy that produced it is reinforced. The model is not taught a stylesheet of reasoning; it discovers which intermediate moves correlate with final success, because intermediate moves are now part of the thing being scored.
This is why the behavior generalizes oddly. On logic, mathematics, and programming — domains where correctness can be checked mechanically — the training signal is dense and the gains are largest. On open-ended prose where no grader exists, the same machinery has far less to bite on. The thinking token is therefore strongest exactly where its output can be audited, a fact that should anchor every expectation about where the technique pays.
03 The bill: reasoning tokens are billed, and they add up
Invisible tokens are not free tokens. API pricing treats generated reasoning as output, at output rates, regardless of whether the user ever reads it. At launch, OpenAI listed o1 at $15 per million input tokens and $60 per million output tokens against GPT-4o's $2.50 and $10 — a sixfold gap on both sides of the ledger. Because a reasoning model may emit thousands of hidden tokens before a visible one, the effective cost of a single answered question can exceed the sticker comparison by multiples that depend entirely on how hard the model finds the problem.
The pricing structure quietly turns difficulty into the unit of accounting. Easy prompts that trigger brief reasoning stay cheap; hard prompts that trigger long deliberation compound. Deployers discovered this empirically when autumn 2024 invoices arrived, and the industry response — reasoning-effort dials, per-request budgets, auto-selecting modes that route trivial queries to non-reasoning models — is an admission that unbounded thinking is a product bug as much as a feature.
04 Latency: when waiting beats guessing
The second currency reasoning spends is time. A non-reasoning model streams a first token in hundreds of milliseconds; a reasoning model may deliberate for ten seconds or several minutes on a hard problem before anything visible arrives. That trade is sometimes obviously worth making — a wrong answer delivered instantly costs more than a right answer delivered late, for engineering work, planning, or anything checked downstream. In a compiler, nobody mourns the milliseconds.
But the same trade is ruinous in a chat window. Interactive assistants live or die on perceived responsiveness, and a model that pauses to think about a greeting has mispriced its own effort. The pattern that emerged through 2025 and 2026 is therefore routing: fast non-reasoning models for conversational traffic, reasoning models invoked selectively for queries detected as hard. Latency management became a portfolio problem — which model answers which question — rather than a property of any single system.
05 Where thinking pays: math, code, planning — and where it doesn't
The clean rule of thumb falls straight out of the training method: reasoning tokens pay where correctness is verifiable. Competitive mathematics, algorithmic coding, multi-step data analysis, formal planning — tasks with checkable end states — see the largest and most reproducible gains, because that is precisely the reward signal the model was optimized against. Benchmark suites in those domains jumped measurably when reasoning models arrived, and the gaps have persisted where tasks remain verifiable.
Where the end state is a matter of judgment — a marketing draft, a summarization, a tone adjustment — extra thinking mostly buys verbosity. The model can deliberate elaborately about a question that never had a checkable answer, and the deliberation cannot improve it the way it improves a proof. Workload fit, not raw capability, determines whether the sixfold price gap is a bargain or a sink, which is why deployment audits now segment traffic by task type before choosing models.
06 Overthinking: wasted compute on prompts that never needed it
A subtle failure mode earned its own name: overthinking. Given a trivial question, an early reasoning model would still enumerate possibilities, second-guess, and wander — burning tokens and seconds on deliberation that changed nothing. Users watched a model spend thirty seconds confirming that two plus two is four. In research evaluations, overthinking shows up as performance that degrades when the model is allowed too many steps, as extended deliberation talks the model out of correct first answers or cycles among them.
The countermeasures define the current product surface: effort settings that cap deliberation length, controllers that stop generation when confidence saturates, and prompt-level instructions to answer directly. Each is a patch on the same underlying problem — the model was trained to use available compute, so unused budget is spent whether or not spending it helps. The engineering question has shifted from whether to think to how much, and that question has no single answer across a mixed workload.
07 Benchmark curves vs real-workload returns
The published test-time compute curves show performance rising with thinking budget, then flattening — a shape that promises returns anywhere on the rising segment. Production experience sharpens that picture: the rising segment is real but task-specific, steepest on verifiable domains and shallowest where judgment rules. A curve measured on competition mathematics says little about a support desk, and budgets tuned on benchmark suites routinely overshoot on messy real traffic where the modal question is easy.
The discipline that has emerged is measurement on your own distribution: sample the workload, replay it at several thinking budgets, and price the accuracy-versus-tokens frontier directly. Organizations that skip this step tend to discover the flattening the hard way, in invoices for deliberation that moved no metric. The curve shape generalizes; its inflection point does not, and the inflection point is where all the money is.
08 Limits: confidence, hallucination, and the inference-scaling ceiling
Thinking longer fixes reasoning failures, not knowledge failures. A reasoning model that lacks the facts still lacks them after ten thousand deliberation tokens — it may simply construct a more elaborate, more confident wrong answer, and the persuasive texture of visible reasoning can make errors harder for users to spot. Test-time compute amplifies whatever the underlying policy knows; it does not add to the inventory. Verification, retrieval, and grounding remain separate problems that deliberation does not absorb.
There is also a ceiling. The scaling curves flatten because deliberation quality is bounded by the base policy — a model can only re-derive so much from what it already represents. Sustained progress then returns to the old axis: better base models, better training, better tools for the deliberation to call. The likely steady state is not one reasoning model but a spectrum — instant cheap answers for the trivial mass of queries, metered thought for the hard tail — with the routing layer, not the model, deciding where your compute budget goes.
References
- OpenAI, Learning to reason with LLMs — o1 announcement and test-time compute curves, https://openai.com/index/learning-to-reason-with-llms/
- OpenAI API pricing (current platform pricing page; launch prices cited as announced September 2024), https://openai.com/api/pricing/
- Wikipedia: Reasoning model — background on reasoning language models, https://en.wikipedia.org/wiki/Reasoning_model
- Source video: https://www.youtube.com/watch?v=tEzs3VHyBDM
By N43 and Hermes AI for DutyStation News.





