Speculative decoding: how AI models answer almost instantly

01The speed problem

Ask a large language model a question and the first word of its answer may appear quickly, but the rest of the reply arrives one token at a time, each token requiring a full pass over the model. This one-token-per-step rhythm, called autoregressive decoding, is what makes streamed replies feel fast on short answers and sluggish on long ones.

The bottleneck is not arithmetic. Modern accelerators are fast at the mathematics of each step; the constraint is that the model's weights must be read out of memory for every single token, and reading terabytes of parameters repeatedly is expensive in time and power. This is called the memory-bound regime.

That single observation drives a whole family of optimizations, and one of the most elegant is speculative decoding, which changes how many tokens a single step produces without changing what the model would have said.

02Draft, then verify

Speculative decoding runs two models instead of one. A small, fast draft model proposes several tokens in a row, cheaply, because it is small enough that reading its weights costs little. Then the large target model, the one whose answers actually matter, checks all proposed tokens in a single forward pass.

The trick is in the check. The target model does not merely approve or reject; it computes the probability it would have assigned to each proposed token at that position. Accepted tokens are kept, and the first token the target model would have chosen differently is substituted, together with a corrected continuation drawn from the target model's own distribution.

The result, as Wikipedia's summary puts it, is that verification preserves the target model's original output distribution. Statistically, the output is indistinguishable from what the big model would have produced alone. Nothing about the model's answers changes; only the work needed to produce them.

03Why the gamble pays

The economics work because verification is nearly free relative to drafting. One forward pass through the large model can evaluate several proposed tokens at once, since attention over a shared prefix amortizes the cost of reading weights across all of them.

Language has structure, and small models learn a lot of it. When the target is generating predictable text, such as boilerplate code, standard phrasing or a list that has already established its pattern, a modest draft model guesses the next tokens correctly most of the time. Those correct guesses ride along in a single verification pass instead of several full steps.

When the draft guesses wrong, the scheme loses a little time, one wasted pass that produced a token or two fewer than a normal step would have. In practice the acceptance rate stays high enough that the trade is strongly positive, and the worst case degrades gracefully toward ordinary speed.

04The acceptance rate

How much you gain depends almost entirely on the acceptance rate, the fraction of drafted tokens the big model agrees with. Code and structured formats draft beautifully; free-form creative prose and surprising turns of phrase draft badly.

Practical systems tune the draft length to the workload. Shorter drafts waste less on rejection when the draft model is off, while longer drafts capture more cheap tokens when it is on, and adaptive schemes adjust the proposal length as they observe recent acceptance.

The quality of the draft model matters more than its raw size. A draft model that shares a family or tokenizer with the target tends to agree more often, which is why deployments often pair a flagship model with a miniature sibling distilled from it.

05Where it runs

Speculative decoding has moved from research papers into production inference stacks over the past few years. IBM's technology channel, whose explainer is featured with this article, presents it as a standard acceleration technique for enterprise inference, and the major open source serving frameworks ship implementations.

The technique is especially attractive where latency is the product. Coding assistants that autocomplete large blocks of highly predictable text, and conversational systems that stream replies, benefit most, because their token patterns let small drafts win consistently.

It also composes well with other inference optimizations. Batch scheduling, quantization and attention optimizations target different parts of the cost structure, and speculative decoding attacks the token-at-a-time rhythm on top of whatever else a serving stack does.

06The cost side

Speed of response is not the only thing that matters; total cost of serving matters too. Speculative decoding reduces latency primarily, and in some configurations it also improves throughput, because accepted tokens are produced per pass rather than per pass each.

The overhead is the draft model itself: extra memory to hold it, extra management complexity, and wasted compute when acceptance is low. For deployments running flat out on saturated hardware, a scheme that spends passes to verify can reduce effective throughput on workloads with poor acceptance rates, which is why it is usually configured selectively.

The deeper point is that inference cost is not a fixed property of a model. It is an engineering target, and techniques like this one are how providers keep serving prices falling even as models grow.

07What it does not change

Speculative decoding does not make a model smarter. The verified output is by construction the same distribution the target model would have produced, which means the technique cannot rescue hallucinations, improve reasoning, or change the substance of a response.

It also does not remove the memory wall. It reduces the number of full passes per token by amortizing, but every accepted token still required its weights to be read. The scheme is a better schedule for the memory traffic, not an elimination of it.

That boundary matters for expectations. Faster responses make long outputs usable and streaming interfaces feel immediate, and they make large models cheaper to operate at scale, but the ceiling on what the model knows and can reason is unchanged.

References