The Draft Model Trick: How Speculative Decoding Beats the One-Token-at-a-Time Wall
Photo: N43 and Hermes AIAutoregressive generation is serial by construction, so parallel hardware mostly idles during decode. Speculative decoding attacks that wall with a draft-and-verify loop that never changes the output.
Source video: What Is LLM Inference? How AI Models Generate Responses · IBM Technology · approximately 136,273 views observed via yt-dlp on 2026-11-20. Independently researched by N43 and Hermes AI.
01 The wall: why generation is serial by design
A large language model answers in the only order it can: one token, then the next, each conditioned on everything before it. That autoregressive loop is not an implementation shortcut — it is the math. The probability of token twenty depends on tokens one through nineteen, so the machine cannot compute them together. A GPU that can schedule tens of thousands of operations in parallel spends a decode step producing a single token, reading the model's entire weight matrix out of memory to do it.
The consequence is an economics inversion that surprises people who learned deep learning through training workloads. Training is arithmetic-starved: the hardware is full. Decode is bandwidth-starved: each step is dominated by the time it takes to stream gigabytes of weights from high-bandwidth memory, and most of the chip's multiplication capacity sits idle while the bytes arrive. The question that defines inference engineering is therefore not 'how do we add more compute' but 'how do we make each expensive weight-read produce more than one token.'
02 The asymmetry that matters: verify in parallel, produce in series
Generation is serial, but verification is not. Checking whether a proposed continuation is correct requires one forward pass over that whole continuation at once — the model scores every proposed token position in a single sweep, the same way it processes a full prompt during prefill. A forward pass that reads the weights once can evaluate five positions as easily as it evaluates one, because the cost is dominated by moving weights, not by the arithmetic done with them.
That asymmetry is the entire basis of speculative decoding. If something cheap can guess what the big model would say — even guessing imperfectly, most of the time — the big model can grade the whole guess in the span of one ordinary decode step. The wall does not come down. It gets walked around: serial production is replaced by parallel auditing of a serial draft, and the expensive resource (the weight read) is amortized across several accepted tokens instead of one.
03 Draft and verify: the mechanics of the speculation loop
The canonical loop works like this. A small draft model — a few hundred million to a few billion parameters — proposes k tokens autoregressively, exactly the way the large model would, but thousands of times cheaper per token. The draft then hands its guess to the target model, which runs one forward pass scoring all k positions simultaneously. Tokens the target model agrees with are accepted; when the target model disagrees, the mismatch point truncates the guess, and the target model's own preferred token is emitted at that position instead.
Then the cycle repeats: the draft starts again from the accepted sequence, proposes another k tokens, and the target grades them in one pass. The target model never runs in autoregressive mode at all during this dance — it only ever executes wide, parallel verification passes. That inversion is the trick: the serial machine is replaced by a fast serial proposer plus a slow parallel grader, and the grader's per-step cost barely changes whether it audits one token or eight.
04 Why the output stays identical: lossless by construction
The part that earns speculative decoding its engineering respect is the guarantee. This is not approximate generation, not a compressed or distilled model whose outputs drift. The modified rejection-sampling scheme used in both the 2022 DeepMind and Google formulations accepts a draft token with a probability rule that — corrected when a rejection occurs, by renormalizing toward the target model's preference — makes the emitted sequence distributionally identical to what the target model would have produced decoding alone. Every token that ships is a token the target model endorsed, or exactly the correction it demanded.
The practical meaning is unusual in performance engineering: the speedup has no quality price. There is no trade-off dial between fast and faithful. The only thing that varies is how often the draft guesses well, which changes how many tokens each verification pass yields — not what any accepted token contains. Teams adopt it precisely because it is one of the few optimizations where the answer to 'does this change the model?' is provably no.
05 Acceptance rates decide everything
The economics of the loop reduce to one number: how often the target model accepts the draft's proposal, averaged across positions. A high acceptance rate means each verification pass yields several tokens for one weight-read; a low rate means the pass yields little and the draft's compute was wasted. The classic speedup model — target cost c for the verification pass, draft cost a fraction of that per token, k draft tokens per round — gives a wallclock speedup that rises steeply with acceptance, flattening only when the draft's own cost starts to eat the gain.
Acceptance is a property of the pairing, not the target alone. A draft distilled from the target, or a small sibling trained on similar data, tracks the target's preferences closely and accepts often. A generic small model guesses the shape of the next token but not the target's exact taste, and acceptance drops. This is why draft selection is now a real engineering discipline: the draft must be cheap enough to pay for itself and accurate enough to keep the verification pass busy.
06 The variant zoo: trees, heads, and n-gram drafting
The base formulation drafts one linear sequence, but the proposal step has since been reimagined several ways. Tree-shaped speculation proposes a branching structure of continuations and verifies it with tree attention, spending a slightly heavier verification pass for a shot at accepting more tokens on uncertain terrain. Medusa-style approaches skip the separate draft model entirely: extra decoding heads attached to the target model itself propose several continuations of the current prefix in one pass, trading a little per-step cost for hardware simplicity.
N-gram and prompt-lookup drafting guess from the input rather than from a model at all: for code, structured text, or editing workloads — where the answer heavily repeats phrases already in the prompt — the next tokens are looked up as continuations of n-grams appearing earlier in the context. These drafts cost nearly nothing and accept spectacularly often on the right workloads, and barely at all on creative generation. The zoo exists because acceptance rate is workload-dependent, and different guessers win on different terrain.
07 What it changes: latency, cost, and where it pays
For interactive products, speculative decoding attacks the metric users actually feel: time per token during streaming. The weight-read that once produced one token now produces several on average, so inter-token gaps compress without any change to the model's answers. The effect on cost per token follows the same arithmetic — fewer target-model passes per emitted token means less bandwidth-bound time billed to the request — which is why serving teams adopted it quickly after the open serving stacks shipped support.
The pattern of adoption reflects its shape: it shines where requests are short-to-medium, batches are small, and interactivity matters — assistants, copilots, agentic loops issuing many small completions. It matters less where enormous batches already fill the GPU, because at high occupancy the verification pass stops being cheap relative to the work the batch was already doing. That boundary — speculative decoding as an interactivity technology, not a universal throughput technology — is the nuance most commentary misses.
08 Limits: wasted compute, second-model memory, and big batches
The costs are real even when the trick works. Rejected drafts burn the draft model's compute for nothing, and the verification pass itself grew — auditing k positions costs more arithmetic than producing one, even while costing roughly the same bytes moved. The draft model also occupies memory on the same accelerator, memory that a serving stack would otherwise hand to KV cache or to a larger batch. On memory-tight deployments, hosting a drafter is a visible opportunity cost, and drafts must justify their residency.
And the win erodes predictably at scale. As batch size grows, a serving system keeps the GPU busy with other sequences' work, so the idleness speculation exploits shrinks; the extra verification arithmetic starts competing with useful batch work, and measured speedups compress toward one. The frontier question is integration: scheduling speculative requests inside continuous batching systems, choosing per-request draft lengths, and mixing n-gram drafters for code workloads with model drafters for prose. The wall is still there. It is simply being negotiated with increasingly specific tools.
References
- Leviathan et al., 'Fast Inference from Transformers via Speculative Decoding,' arXiv:2211.17192 — the canonical speculative decoding formulation and lossless guarantee.
- Chen et al., 'Accelerating Large Language Model Decoding with Speculative Sampling,' arXiv:2302.01318 — the parallel rejection-sampling formulation.
- Wikipedia: Large language model — background on large language model inference.
- Source video: What Is LLM Inference? How AI Models Generate Responses (IBM Technology, ~136K views, observed 2026-11-20).
By N43 and Hermes AI for DutyStation News.





