The KV Cache: The Memory Trick That Makes AI Feel Instant

01 Attention's Expensive Secret

Transformers generate text one token at a time, and every new token must attend to every token that came before it. Attention works by comparing the new token's query vector against the key vectors of all previous positions, then combining the resulting weights with those positions' value vectors. The natural, naive way to do this is to recompute the keys and values for the entire prefix on every single generation step.

That recomputation is not just wasted arithmetic; it is wasted arithmetic that grows with the square of the context. If a model has already processed a thousand tokens, producing token 1,001 means running the full key and value projections for all thousand earlier positions again, purely to obtain results that were already computed a step earlier. The same matmuls, against the same inputs, yielding the same answers, hundreds of times over.

This is the bottleneck the key-value cache exists to remove. The insight is almost embarrassingly simple: keys and values for past tokens never change. Once computed, they can be stored and reused. The general principle is an old one in computing, caching a result you will need again rather than recomputing it, and it is the same idea behind memoization in any algorithm. Applied to transformers, it turns a quadratic chore into something close to linear work per step.

02 Caching Keys and Values, Token by Token

During generation, each new token passes through the model once. Its query, key and value vectors are computed for that token only, while the keys and values for all previous tokens are read from the cache. The new token's key and value entries are then appended, so the next step finds them waiting. Every transformer layer keeps its own separate cache, since each layer's keys and values live in its own representational space.

The effect on cost per token is dramatic. Without the cache, generating a sequence of length N means recomputing attention over the growing prefix at every step, so total work grows with the square of N. With the cache, each step only projects one new token and reads stored history, so per-token cost stays nearly flat as the response grows. The same property explains why the first token of a reply, the prefill over the prompt, is slower than every token after it: prefill computes and caches the whole prompt in one pass before fast decoding begins.

This is the quiet mechanism behind the "streaming" feel of a chatbot. Each token after the first arrives at a much lower cost than the prompt processing that preceded it, and the visible typing effect is not a UI trick but a direct readout of cache economics. The cache converts the transformer from a machine that relives its entire history on every word into one that merely glances back at it.

03 How the Cache Grows With Context

The cache is not free storage; it is GPU memory, the scarcest resource on an inference card. Its size follows a standard formula: two tensors (keys and values), times the number of layers, times the number of attention heads used for keys and values, times the head dimension, times the bytes per element, times the context length. With fp16 weights, a 7B-class model with grouped-query attention stores about 128 kilobytes per token. Doubling the context doubles the cache, exactly, with no plateau.

The arithmetic becomes sobering at long context. That same 7B-class GQA model holds roughly 1 gibibyte of cache at an 8,000-token context, 2 at 16,000, 4 at 32,000 and 8 at 64,000, all per request, before the model's own weights are counted. A card with 80 gigabytes of memory might fit the 14 or so gigabytes of fp16 weights comfortably, then discover that a handful of long-context users quietly consume everything left.

Batch size multiplies the pain. Serving ten concurrent users at the same context means ten independent caches, since no two conversations share history. Operators therefore cap the product of batch size and context length, trade one against the other, or offload cache to CPU memory at the cost of transfer bandwidth. Every long-context feature a lab advertises is, behind the scenes, a negotiation with this growth curve.

04 GQA and MQA: Shrinking the Bill

Since cache pressure scales with the number of key-value heads, the obvious fix is to use fewer of them. Multi-query attention takes this to the limit: many query heads share a single key-value head per layer, cutting cache size by a factor equal to the query-head count. Grouped-query attention is the gentler middle ground, sharing each key-value head among a small group of query heads, and it is the approach used in widely deployed models including Llama 2 70B and its successors.

The trade-off is quality versus memory. Full multi-head attention gives every query head its own keys and values, which is expressive but expensive to cache. Sharing heads saves enormous memory, but forces unrelated query heads to attend through the same key-value lens, which can cost a little accuracy. The trend across recent model releases is unmistakable: nearly every new large model ships with GQA or MQA, accepting a small quality risk to buy back gigabytes of cache.

The numbers make the reasoning plain. At 8,000-token context and fp16, a 13B-class model with full multi-head attention needs roughly 12.5 gibibytes of cache, while a 70B model with full multi-head attention demands about 20 gibibytes before its 140 gibibytes of weights even load. The same 70B with multi-query attention drops to roughly 0.3 gibibytes, a 64-fold cut. Modern architecture choices are, to a first approximation, cache-pressure management.

05 The Speed-Memory Tradeoff in Practice

Every inference deployment eventually faces the same ledger. The cache buys speed by spending memory, and memory spent on cache cannot hold weights, activations or additional users. A serving system tuned for latency keeps caches resident and batches small; one tuned for throughput evicts caches aggressively, reuses memory for larger batches and accepts recomputation when a request resumes.

Techniques like paged attention, cache quantization and prefix sharing exist to bend this curve. Paged attention manages cache in fixed-size blocks like operating-system memory pages, reducing fragmentation; quantizing cached keys and values to 8-bit or 4-bit shrinks them severalfold at a small quality cost; prefix sharing lets many requests that begin with the same system prompt reuse one cached copy. Each is a variation on the same theme: pay less memory per cached token.

What no technique can do is abolish the trade, because it is structural. Caching only helps if the stored data is actually reused, and attention reuses everything, every step, which is exactly why the KV cache works and exactly why it balloons. The engineering question is never whether to cache but how much to cache, in what precision, and for how many simultaneous users.

06 Caching as a Design Philosophy

Placed in the longer history of computing, the KV cache is unglamorous. Caches have been a foundational idea since memory hierarchies existed: CPU register files, L1 and L2 caches, and web caches all exploit the same observation that recomputation costs more than storage. What the transformer adds is scale; the KV cache is arguably the first cache whose size becomes a headline product feature, driving GPU purchases, dictating context windows and shaping model architecture itself.

That influence runs upstream into model design. The choice between full multi-head, grouped-query and multi-query attention is made before a model ever trains, and it is made with cache economics in mind, a rare case where serving-time memory behavior shapes research-time architecture. Sliding-window and linear-attention variants go further, replacing exact cached history with compressed or bounded alternatives, trading recall fidelity for a cache that does not grow with context at all.

The lesson generalizes beyond AI. The KV cache is a reminder that in modern computing, moving and storing data is often the dominant cost, not arithmetic. A transformer's floating-point units are rarely the constraint; the bandwidth to feed them is. Systems that feel instant are usually systems that arranged, well in advance, to never compute the same thing twice.

07 The Takeaway for Anyone Watching AI

For users, the KV cache explains a familiar asymmetry: the first word of a long answer takes noticeably longer than the rest, and a conversation that has grown long eventually feels heavier. Neither effect is the model "thinking"; both are cache economics, prefill cost on the front end and accumulated context on the back.

For operators, the cache is the central planning number. Context-window claims are bounded by memory, concurrency is bounded by the same memory, and precision choices for cache, weights and activations determine how much of a card's capacity is actually billable. Capacity planning for inference is, more than anything else, a projection of the growth curve in section 03 multiplied by expected batch size.

And for anyone evaluating AI infrastructure generally, the KV cache is the right lens through which to read vendor claims. When a new model boasts a longer context window, ask what it costs in memory per user. When a new accelerator boasts bandwidth, ask how it feeds attention. The chips and models may change, but the underlying bargain, trading storage for recomputation, predates the transformer and will outlast it.