Attention, Explained: The Mechanism Behind Every Modern LLM

01 Why Neural Networks Needed Attention

For roughly three decades, the dominant way to process language was the recurrent neural network, or RNN. An RNN reads a sentence the way a driver reads a toll road: one word at a time, updating a running internal summary as it goes. That summary — a single fixed-size vector of numbers — had to hold everything the model had seen so far. By the end of a long paragraph, the earliest words had been overwritten so many times that their influence on the output was effectively gone.

The failure mode had a name: context loss. Ask an RNN-based translator what a pronoun referred to ten sentences back, and it guessed. The fix arrived through attention, first popularized in machine translation around 2014 and 2015: instead of relying on one compressed summary, the model could look back at every input word and learn, for each output step, exactly which ones mattered. Relevance, in other words, became something a network could measure rather than remember.

02 How Self-Attention Works: Queries, Keys, Values

Self-attention applies that look-back trick inside a single passage of text. Every token — roughly, a word or word-fragment — is first converted to a vector. From that vector the model derives three smaller ones: a query, which asks what am I looking for; a key, which advertises what do I contain; and a value, which holds the actual information to share. Each token's query is compared against every other token's key with a dot product, producing a relevance score for every pair of words in the passage.

Those scores pass through a softmax function, which converts them into weights that sum to one. The new representation of each word is then a weighted average of all the value vectors, dominated by whichever words scored highest. In the sentence 'The animal didn't cross the street because it was too tired,' the token 'it' scores 'animal' far above 'street,' and its meaning updates accordingly. Modern models run many such comparisons in parallel — multi-head attention — so different heads track grammar, topics and references at once.

03 From Attention to the Transformer

The 2017 paper 'Attention Is All You Need' by Vaswani and colleagues at Google removed the recurrent layer entirely and kept only attention, stacked in blocks with small feed-forward networks between them. Because every token's representation is computed from scratch in the same pass, with no sequential dependency from one word to the next, training could be parallelized across thousands of processors. Recurrent models were structurally incapable of that; the transformer turned model-building into a hardware problem, and hardware was getting cheaper fast.

One thing was lost in the trade: word order. Self-attention on its own treats a sentence as an unordered bag of tokens, so the transformer injects order explicitly through positional encodings — numerical patterns added to each token's vector that tell the model where it sits in the sequence. The span a model can contextualize in one pass is its context window, the number that has grown, as Figure 1 shows, from 512 tokens in 2018 to a measured 1,000,000 in the 2025 Claude generation.

04 Scaling Up: Tokens, Pretraining and Emergence

A large language model is a transformer trained on vast amounts of text to do one thing: predict the next token. Given 'The capital of France is,' the model assigns probabilities to every possible continuation, and 'Paris' wins. Pretraining on web pages, books and code — trillions of tokens by the estimates published alongside the major models — forces the network to internalize grammar, facts, reasoning patterns and a surprising amount of world knowledge, simply because accurate prediction requires it.

Scale did something unexpected. At certain combinations of model size, data and compute, abilities appeared that were never explicitly programmed: learning from examples placed in the prompt, multi-step reasoning, competent translation and summarization. This is why the same basic architecture underlies ChatGPT, Claude, Gemini, Grok, Qwen and DeepSeek, as the Wikipedia overview of large language models documents: the differences between assistants lie mostly in data, training method and fine-tuning, not in the attention mechanism itself.

05 Where Attention Struggles

Attention's costs grow quickly. Because every token scores every other token, doubling the sequence length roughly quadruples the computation — the illustrative curve in Figure 2 below. Context windows are therefore bounded by memory and money as much as by architecture, and quality inside a very long window can be uneven: models sometimes lose track of details buried in the middle of a document even when the nominal limit is measured in the hundreds of thousands of tokens.

Fluency creates the second weakness: hallucination. A model optimized to produce the most plausible next token has no built-in sense of what is true, so it can state invented facts in the same confident tone as real ones. The standard mitigation is retrieval — handing the model documents to ground its answer — which is why file uploads and search integration became default assistant features. Hallucination rates have fallen, but no vendor claims zero, and careful users verify anything that matters.

06 The Efficiency Frontier: KV Caches, Distillation, Inference Economics

Serving attention economically is an engineering discipline of its own. The dominant trick is the KV cache: during generation, the keys and values of already-processed tokens are stored in memory, so each new token only computes attention against the cache instead of re-reading the whole sequence. That converts per-token work from quadratic to roughly linear, at the price of memory — cached context is a large part of why long-conversation requests cost more to serve than short ones.

Around the cache sits the rest of the toolkit: quantization shrinks the numbers that make up a model so more of it fits on each chip; distillation trains a smaller student model to mimic a larger teacher, trading a measured amount of accuracy for a large cut in cost; batching lets one processor serve many users at once. None of these change what attention is — they change what a token costs, and cost increasingly decides which features a product can afford to ship.

07 What to Watch in 2027

Three fronts look decisive heading into 2027. First, context: the measured jump to 1,000,000-token windows in the 2025 Claude generation reframes what counts as a long document, and whether accuracy holds across that whole span matters more than the headline number. Second, architecture: sub-quadratic attention and hybrid designs that mix attention layers with recurrent-style state aim to break the cost curve in Figure 2 without giving up quality.

Third, agents. When a model runs tools, reads files and acts over hours rather than answering once, the context window becomes working memory and attention efficiency becomes the binding constraint on how long a task can run. The indicators to watch through 2027 are price per million tokens, published long-context accuracy results and agentic benchmarks; together they will show whether the transformer's second decade looks like its first.