How Transformers Learn to Pay Attention

01The old bottleneck was sequence order

Before transformers, language models commonly relied on recurrent networks that processed tokens in sequence. That made long-range dependencies expensive: information had to travel step by step through hidden states. The 2017 paper Attention Is All You Need removed recurrence from the core architecture. Self-attention lets every token read from every other token in the same window, while positional information restores the fact that “dog bites man” is not the same as “man bites dog.”

The result is not magic memory. It is a highly parallel matrix computation. During training, many positions can be processed together, which makes modern accelerators far more useful than a strictly serial loop.

02Query, key, value: three roles for one token

For each token representation, a learned projection produces a query, a key, and a value. A query asks what this position needs; keys advertise what other positions contain; values carry the information that will be mixed into the answer. The attention score compares a query with keys using a dot product, scales by the key dimension, and normalizes with softmax.

In compact notation, the operation is softmax(QKᵀ / √d_k)V. The square-root term keeps dot products from becoming too large as vectors grow. Softmax turns scores into weights that sum to one, creating a differentiable information-routing system rather than a hard lookup table.

03Why multiple heads matter

A single attention map has one way to connect positions. Multi-head attention runs several smaller projections in parallel, allowing different heads to specialize in different relationships: a syntactic dependency, a nearby phrase, a repeated name, or a position in a list. The heads are concatenated and projected back into the model width.

These heads are not guaranteed to correspond neatly to human concepts. They are learned computational subspaces. Some become interpretable; others work in combination with feed-forward layers and residual streams. The useful unit is the entire network, not a single colorful attention visualization.

04The transformer block is a repeated circuit

A practical transformer layer alternates attention with a position-wise feed-forward network. Residual connections let a layer add a correction to the running representation, while normalization stabilizes the scale of activations. Stack enough layers and the model repeatedly revises each token’s representation: first local context, then syntax, then increasingly abstract patterns.

Decoder-only models use a causal mask so a position cannot inspect future tokens during next-token training. Encoder models such as BERT can use bidirectional context for representation learning. Encoder–decoder designs use cross-attention to connect an output sequence to an input sequence.

05Scaling made the architecture a platform

The transformer paper began as a machine-translation architecture, but the same primitives generalize to text, images, audio, robotics, and multimodal systems. The trade-off is attention’s quadratic relationship with sequence length in its straightforward form: doubling the window can multiply the pairwise score work by roughly four.

That pressure produced engineering advances such as FlashAttention, key–value caching, multi-query attention, sparse patterns, and alternative positional encodings. These are not cosmetic optimizations. They determine which context lengths, latency targets, and memory budgets are economically possible.

06What attention does—and does not—explain

Attention explains a central mechanism for moving information, but it does not by itself explain factual reliability, reasoning, or consciousness. The network also contains learned embeddings, feed-forward transformations, normalization, output probabilities, and a training objective. Its abilities emerge from the interaction of architecture, data, optimization, and scale.

The clean mental model is therefore modest and powerful: a transformer is a stack of differentiable routing-and-rewriting blocks. Attention decides which representations can influence one another; the rest of the block transforms that mixture. Understanding the circuit is the beginning of understanding the model, not the end.