How Transformers and Attention Work: The Architecture Behind Every Modern LLM

01 From Recurrent Bottlenecks to a Parallel Revolution

For a decade before 2017, sequence modeling belonged to recurrent neural networks and their gated descendants, the LSTM and GRU. These systems read text one token at a time, folding everything they had seen into a single fixed-size hidden state. That hidden state is a lossy bottleneck: by the time a recurrent network reaches the end of a long paragraph, the information at the beginning has usually been overwritten or blurred beyond use. Gradients flowing backward through hundreds of timesteps vanish or explode, which made long-range dependencies, the kind a pronoun and its distant antecedent demand, structurally painful to learn.

The second problem was economic rather than mathematical. Recurrence is inherently sequential: a sequence of a thousand tokens requires a thousand dependent steps no matter how many processors you own, so recurrent training left GPUs sitting idle. The 2017 Google research paper "Attention Is All You Need" removed recurrence from the picture entirely. In the architecture it introduced, the transformer, every token in a sequence can consult every other token in a single parallel step, and the entire network becomes a stack of large matrix multiplications, precisely the workload that modern accelerators were designed to saturate. That single design decision explains most of what happened next.

02 Tokens and Embeddings: Language as Geometry

Before attention can operate, language has to become numbers. A tokenizer first splits raw text into subword units drawn from a fixed vocabulary, typically tens of thousands of entries. Subword schemes such as byte-pair encoding handle both common words and rare ones gracefully: "unbelievable" might survive as a single token while "antidisestablishmentarianism" breaks into several pieces. Each token then maps to a point in a high-dimensional vector space through a learned embedding table. In the original transformer this space had 512 dimensions; frontier models of the mid-2020s work in spaces of 8,000 dimensions or more.

The geometry of this space is learned, not designed, and it is remarkably semantic: similar concepts end up near one another, and directions in the space encode relationships such as tense or plurality. Crucially, the embedding assigned to a token is only its starting position. As the input passes through the network's layers, each token's vector is progressively rewritten to reflect its context, so the same word can carry different internal representations in different sentences. Attention is the mechanism that performs that rewriting.

03 Attention as a Soft Lookup Table

Mechanically, attention is easiest to understand as a dictionary lookup with weights instead of hard matches. Each token's vector is projected three ways, producing a query, a key, and a value. The query encodes what the current token is looking for; the key advertises what each token in the sequence contains; the value holds the information actually to be passed along. A token takes the dot product of its query with every key, producing relevance scores, which a softmax converts into positive weights that sum to one. The output is the weighted average of all the value vectors, weighted by how relevant each token judged the others to be.

Two properties make this powerful. It is a soft lookup: rather than retrieving exactly one entry the way a hash table does, attention blends information from every position in proportion to relevance, so a pronoun can pull context from a noun that appeared forty tokens earlier while still hearing from its nearer neighbors. And it is fully differentiable, so the entire mapping, queries, keys, and values alike, is learned end to end from data. In his visual talk for TNG Big Tech Day 2024, Grant Sanderson builds exactly this intuition, animating how queries and keys interact so that grammatical structure and long-range reference emerge from learned dot products rather than hand-coded rules.

04 Multi-Head Attention and the Feed-Forward Stack

A single attention operation forces a model to summarize everything a token needs from its context into one weighted average, which is a surprisingly blunt instrument. The transformer's answer is multi-head attention: run many small attention operations in parallel, each in a lower-dimensional subspace, then concatenate their outputs and project them back. With eight heads, as in the original architecture, one head might learn to track subject-verb agreement, another to bind pronouns to names, another to attend to adjacent tokens, and none of these learned specializations needs to be specified in advance. They fall out of training.

Attention does not act alone. Each transformer block pairs its attention layer with a position-wise feed-forward network, which expands each token's vector, typically by a factor of four, applies a nonlinearity, and compresses it back. Research has suggested these feed-forward layers act as associative memories holding much of a model's factual knowledge, while attention moves information between positions. Residual connections around each sublayer and layer normalization keep gradients healthy through depth. Stack these blocks twelve high and you have GPT-2 small; stack them a hundred or more high, at far greater width, and you have the shape of a frontier model.

05 Position Encoding: Teaching Order to a Memoryless Machine

Attention has a hidden flaw that follows directly from its design: it is permutation-invariant. Since every token attends to every other token symmetrically, the mechanism by itself has no way to distinguish "dog bites man" from "man bites dog" or to know which word came first. The fix is to inject position information explicitly. The original transformer added sinusoidal waves of different frequencies to each token's embedding, a mathematical fingerprint of its index in the sequence. The GPT family instead used learned absolute positional embeddings, one vector per slot, trained like any other parameter.

Modern models have largely moved to relative schemes, most prominently rotary embeddings, which rotate query and key vectors by position-dependent angles so that attention scores depend on the distance between tokens rather than their absolute slots. Relative encodings generalize better to lengths never seen in training, which is why the long-context releases of recent years lean on rotary variants such as position interpolation and NTK-aware scaling. A significant fraction of the "context window" progress advertised between 2023 and 2026 is not a change to attention itself but a smarter way of stamping position onto it.

06 Next-Token Prediction and the Economics of Scale

The training objective that turns this architecture into a working model is almost embarrassingly simple: given a sequence of tokens, predict the next one. The model outputs a probability distribution over its entire vocabulary at every position, and training measures the cross-entropy loss of the true next token, averaged across trillions of tokens of text and code. There is no explicit grammar module, no hand-labeled syntax, and no curriculum beyond the data itself. The same objective trains a 117-million-parameter toy and a frontier system; only the scale changes.

What made that simplicity decisive was the discovery that the loss obeys scaling laws: smooth, predictable power-law relationships between validation loss and the amount of data, parameters, and compute, documented first by OpenAI researchers in 2020 and refined by DeepMind's Chinchilla analysis in 2022. If loss falls predictably as you scale, then capability becomes a budgeting problem, and the industry responded by scaling relentlessly, as the parameter history below shows.

Parameter counts eventually stopped being the headline metric. Sparse mixture-of-experts designs, popularized by Mixtral and widely believed to underpin GPT-4, activate only a fraction of a model's weights per token, so total parameters and active parameters diverged sharply. Trackers such as Epoch AI now follow training compute as the more honest yardstick, and it has grown roughly fourfold to fivefold per year, a pace that has carried straight through the releases of the mid-2020s.

07 The Quadratic Wall: Context Windows in 2026

Attention's defining strength, letting every token consult every other token, is also its defining cost. A sequence of n tokens requires n-squared relevance comparisons per attention head, and the memory needed to store the intermediate scores grows the same way. The numbers escalate quickly: a 1,000-token context implies about a million token-pair comparisons, while a 128,000-token context implies roughly 16.4 billion, as the illustrative chart below shows. This quadratic scaling, not parameter count, is what makes long context expensive, and it is the reason the attention matrix is the first place engineers look when they need to cut costs.

The engineering responses have been substantial. FlashAttention recomputes attention in tiles so the full matrix is never materialized in memory, turning a memory bottleneck into a compute-dense stream. Sliding-window and sparse attention variants, as used in Mistral-class and many open models, restrict each token to a local neighborhood plus a handful of global tokens, sacrificing some connectivity for large constant-factor savings. Quantized key-value caches, cache compression, and hardware-aware kernels each claw back further multiples. Meanwhile, providers kept stretching position encodings so that models trained at one length could be served at ten or a hundred times it.

The result is a decade of context-window growth that outpaces nearly any other metric in the field, from the 1,024 tokens of GPT-2 to the million-token class introduced by Gemini 1.5 Pro and since matched or exceeded by several frontier systems.

A caveat belongs next to every one of those numbers: a stated context window is a capacity, not a guarantee of quality. Independent evaluations of long-context recall have repeatedly found that models begin missing information well before their nominal limit, particularly for items buried in the middle of a long prompt. As of 2026, the transformer has largely stopped being the bottleneck; attention is well understood, its kernels are heavily optimized, and its variants are catalogued. The binding constraints are the ones attention cannot fix, data quality, training compute, and the economics of inference, which is why the most consequential debates in the field have moved from architecture to scale. Until something displaces it, the attention diagram that Sanderson sketches on stage remains the load-bearing picture of the entire industry.