What Is a Context Window? The Working Memory Behind Every LLM Answer

01 The Memory That Is Not Memory

A large language model has no memory in the human sense. Between one conversation and the next, its weights stay frozen and untouched. What feels like recall is something much more mechanical: everything the model knows about your specific exchange is physically present in a single block of text called the context window. When the model responds, it can only condition its output on what sits inside that block, plus the general patterns baked into its training.

Think of the context window as working memory rather than long-term memory. A person can recall a phone number long enough to dial it but will not remember it next week unless it is written down. The model behaves the same way. If a fact, a correction, or an instruction is no longer inside the current window, it effectively never happened. This is why a chatbot can seem inconsistent over a very long conversation: the earliest turns may have quietly slid out of the window, and nothing in the model protests when they do. The window is the entire universe of conversational evidence the model can draw on, and it is bounded, finite, and shared by everything you send.

02 Tokens, Not Characters

Context windows are not measured in words or characters but in tokens, the sub-word units that tokenizer algorithms chop text into before the model ever sees it. A common rule of thumb in English is that one token corresponds to roughly four characters, or about three-quarters of a word, though the ratio shifts with language and formatting. Unusual spellings, code, and non-English scripts often expand into more tokens than plain prose.

The practical consequence is that users chronically misjudge capacity. A window of 128,000 tokens is not 128,000 words; it is closer to 96,000 words of plain English, and considerably less if the text is dense with code, URLs, or tables. Tokenization also explains quirks that look like reasoning failures but are really accounting artifacts: a model may count letters wrong inside a word because it never saw individual letters, only token fragments. When planning what fits into a session, whether that is a codebase, a legal filing, or an archive of emails, the token count of the material, not its page count, is the number that governs what the model can actually hold.

03 The Quadratic Price of Attention

Why not simply make the window enormous and be done with it? The core obstacle is the transformer's self-attention mechanism. Standard attention asks, for every pair of positions in the input, how much one should influence the other. Double the sequence length and the number of pairwise comparisons grows roughly fourfold. Compute and memory for attention therefore scale with the square of the context length, a relationship that turns linear-seeming increases in window size into brutal costs in training time, inference latency, and GPU memory.

This quadratic curve is the central economic fact of context engineering. Every additional token a model attends over makes every other token marginally more expensive to process. Providers price long-context requests accordingly, and researchers spend enormous effort on techniques that bend the curve: sparse attention patterns that skip unlikely pairs, sliding windows that focus compute on neighboring tokens, and hybrid architectures that reserve full attention for a subset of positions. The chart below illustrates the shape of the cost curve; the exact slope varies by architecture, but the accelerating growth is the universal feature.

04 How Models Grew From 1K to a Million Tokens

The growth of context windows across model generations has been one of the steepest capability curves in the field. GPT-2, released in 2019, operated over roughly 1,024 tokens. GPT-3 raised this to about 2,049 tokens with its tokenizer. GPT-4 shipped in 2023 with an 8K window and a 32K variant, and successive frontier releases through 2024 and beyond announced windows in the hundreds of thousands, with flagship systems advertising one million to two million tokens of context. These are announced, provider-reported figures rather than independently audited measurements, but the direction is unmistakable.

05 The Engineering That Stretches the Window

Extending context is not a single invention but a stack of techniques. Rotary position embedding, known as RoPE, encodes where each token sits using rotations rather than fixed positions. Researchers found they could stretch or rescale the rotation frequencies, a family of tricks generally called RoPE scaling, to teach a model trained at short lengths to operate at far longer ones without retraining from scratch. Other approaches include sparse and sliding-window attention, which prune the all-pairs comparison to keep cost manageable, and hybrid layers that interleave cheap local attention with a few expensive global layers.

Retrieval takes the opposite approach: instead of making the window bigger, it makes the window smarter. A retrieval system stores documents outside the model and injects only the passages judged relevant for the current question. The model reads a few hundred well-chosen tokens instead of a million indifferent ones. In practice, the frontier is a hybrid. Long windows handle coherent, session-scale material such as an entire code file or a single long report, while retrieval fetches the wider world on demand. The best systems increasingly look like compilers for context, deciding what the window should contain, rather than engines with a fixed fuel tank.

06 Lost in the Middle

A larger window is not the same as a larger useful memory. A widely cited 2023 study, "Lost in the Middle," by researchers from Taurus Liu and colleagues at the University of Edinburgh and partners including Cohere and Stanford-affiliated authors, found that retrieval-augmented models perform markedly better when the needed information sits at the beginning or end of the context, and degrade when it sits buried in the middle, even at context lengths the model nominally supports. The effect resembles a person skimming the first and last paragraphs of a long report while glossing over the center.

This finding has practical teeth. If you paste a 300-page document and ask about a clause on page 150, you may get a worse answer than if you had extracted that section and supplied it alone. The model sees the tokens but does not weight them equally, and effective use of long context depends heavily on how material is arranged. The industry response has been a mixture of architectural improvements, retrieval strategies that deliberately order evidence by relevance, and pressure tests like the Needle-in-a-Haystack evaluations that probe whether a model can actually find a fact buried in a long window. Marketing a million tokens is easy; making all million of them equally available to the model is still open research.

07 What Long Context Changes for Users

For everyday users, the practical meaning of long context is that the unit of work grows. In coding, whole-repository analysis becomes possible: instead of pasting one function at a time, an assistant can read an entire project's worth of files, trace a variable across modules, and propose a change that respects conventions defined elsewhere in the codebase. In document work, a model can hold an entire contract, filing, or technical standard in view and answer comparative questions that span hundreds of pages without losing the thread.

The same expansion is reshaping media understanding. Multi-hour video understanding, where a system ingests a lecture, deposition, or broadcast end to end and answers questions about any moment in it, is essentially a long-context problem with a transcription front end. The frontier use cases all follow the same pattern: they convert messy, human-scale corpora into token spans and ask the window to hold them. The limits users actually feel are now less about raw capacity and more about reliability inside that capacity: whether the model uses the middle of the window as well as the edges, whether it resists distraction from irrelevant passages, and whether the cost of a million-token request is worth the answer it produces. The context window has stopped being a footnote of model specs and become the main thing engineers design around.