Transformers: The Neural Architecture Powering Every Modern LLM

Figure 1: Parameter growth across major transformer models. Values for GPT-4, Claude 4, and GPT-5 are estimated from technical reports and analyst projections. Y-axis is compressed for readability.

01 The Problem Transformers Solved

Before 2017, natural language processing relied on recurrent neural networks (RNNs) and long short-term memory networks (LSTMs). These architectures processed text sequentially, one word at a time, maintaining a hidden state that was supposed to carry information from earlier in the sentence to later positions. In practice, this meant that by the time an RNN reached the end of a paragraph, the signal from the opening sentence had often decayed to near-noise. The network could not reliably connect a pronoun at position forty to its antecedent at position five.

This vanishing gradient problem was not merely a technical inconvenience. It placed a hard ceiling on the length of text that neural networks could effectively process. Training was also slow because the sequential dependency meant that GPU parallelism, the single greatest hardware advantage available to deep learning, could not be applied to the time dimension. You could parallelize across a batch of sentences, but within each sentence, every step waited on the previous one.

The transformer architecture, introduced by Ashish Vaswani and colleagues at Google in the 2017 paper "Attention Is All You Need," eliminated recurrence entirely. Instead of processing tokens one at a time, the transformer ingests an entire sequence simultaneously and uses a mechanism called self-attention to let every token look at every other token in parallel. This single design decision removed the sequential bottleneck and unlocked the scale that defines modern AI.

02 Self-Attention: The Core Mechanism

Self-attention is the operation that gives transformers their power. For each token in an input sequence, the model computes three vectors: a query, a key, and a value. The query represents what the token is looking for. The key represents what the token offers. The value is the information the token actually passes along. The attention score between any two tokens is the dot product of one token's query with the other's key, scaled by the square root of the dimension size, then passed through a softmax function to produce a probability distribution.

What this means in practice is that every token can attend to every other token with a weight that is learned during training. In a sentence like "The bank by the river was muddy," the word "bank" can learn to attend strongly to "river" rather than to unrelated words, resolving the ambiguity that would stump a sequential model. The attention weights are not hardcoded but are learned from data, which means the model discovers linguistic relationships through exposure to billions of examples.

Multi-head attention extends this idea by running several attention computations in parallel, each with its own learned query, key, and value projections. Different heads can specialize in different relationships: one might track syntactic dependencies, another might follow coreference chains, a third might detect semantic similarity. The outputs of all heads are concatenated and projected back to the model dimension. This multi-head structure is what allows a single transformer layer to capture multiple types of relationships simultaneously.

03 Positional Encoding: Giving Attention a Sense of Order

Because self-attention is fundamentally a set operation, it is permutation-invariant: it does not inherently know which token comes first. This is a problem for language, where word order carries meaning. "The dog bit the man" and "The man bit the dog" contain the same words but describe very different events.

The original transformer solved this with sinusoidal positional encodings, which add a unique pattern to each position's embedding based on sine and cosine functions of varying frequencies. These patterns allow the model to learn relative positions without hardcoding absolute positions. Later architectures have adopted learned positional embeddings, rotary positional embeddings (RoPE), and algebraic position schemes like ALiBi, each offering different generalization properties to longer sequences than those seen during training.

The choice of positional encoding has significant practical consequences. Models using RoPE, such as LLaMA and its descendants, can extend their context window through interpolation techniques without retraining from scratch. Models using absolute positional embeddings typically need to be retrained or fine-tuned to handle longer sequences. This seemingly minor design choice determines whether a model can process a 128,000-token document or is stuck at 4,096.

04 The Encoder-Decoder Split

The original transformer was an encoder-decoder architecture designed for translation: the encoder processed the source language sentence, and the decoder generated the target language sentence auto-regressively. The encoder used bidirectional self-attention, meaning every token could attend to every other token in both directions. The decoder used masked self-attention, meaning each position could only attend to earlier positions, preventing the model from peeking at future tokens during generation.

The major model families have since diverged. BERT, introduced by Google in 2018, is encoder-only and bidirectional, making it excellent for understanding tasks like classification and named entity recognition but incapable of text generation. GPT and its descendants are decoder-only, using masked self-attention to predict the next token in a sequence. This auto-regressive approach has proven remarkably general: next-token prediction can be framed as a task that encompasses translation, summarization, question answering, and creative writing, all without architectural changes.

The decoder-only design has come to dominate the LLM landscape. Every model in the GPT, Claude, Gemini, LLaMA, and Mistral families uses this architecture. The encoder-decoder design persists in specialized translation models like Google's T5 and in sequence-to-sequence tasks, but for general-purpose language models, the decoder-only transformer has won decisively. The reason is partly empirical and partly practical: decoder-only models are simpler to train, easier to scale, and the auto-regressive objective is a natural fit for the compute-intensive pretraining phase.

Figure 2: Estimated training compute for major transformer models. Values from Epoch AI compute database and published technical reports. Logarithmic growth in compute reflects scaling law findings.

05 Scaling Laws and the Race for Parameters

In 2020, Jared Kaplan and colleagues at OpenAI published scaling laws showing that model performance, measured by loss on held-out data, improves as a power law of three factors: model parameters, dataset size, and training compute. Crucially, these laws suggested that simply making models bigger, training them on more data, and spending more compute would yield predictable improvements. This finding was the theoretical justification for the race from GPT-2's 1.5 billion parameters to GPT-3's 175 billion and beyond.

DeepMind's Chinchilla paper in 2022 refined these scaling laws, showing that most models at the time were significantly undertrained: they had too many parameters relative to the amount of data they were trained on. The optimal ratio was approximately 20 tokens of training data per parameter. This insight meant that a 70-billion-parameter model trained on 1.4 trillion tokens could match or exceed a 175-billion-parameter model trained on 300 billion tokens. The field responded by investing heavily in data quality and quantity, not just parameter counts.

By 2026, the frontier has shifted to multi-trillion-parameter models trained on tens of trillions of tokens, with training runs consuming tens of thousands of GPUs for months at a time. The compute required for frontier models has grown by approximately four orders of magnitude since GPT-2 in 2019. This exponential trend has driven NVIDIA's data center revenue from under 3 billion dollars in 2020 to over 115 billion in 2025, making GPU manufacturing capacity a strategic priority for nation-states.

06 Limitations and Open Problems

The transformer is not without limitations. The self-attention mechanism has quadratic memory complexity in sequence length: doubling the context window quadruples the memory required for the attention matrix. This is why context windows, despite steady growth from 2,000 tokens in early GPT-3.5 to over 200,000 tokens in Claude 4 and Gemini 2, remain a hard engineering challenge rather than a solved problem. Researchers have developed linear attention variants, sparse attention patterns, and chunked attention to reduce this cost, but each comes with quality trade-offs.

Transformers also lack an inherent notion of persistent memory across conversations. The model's parameters are frozen after training, and any context must be supplied in the prompt. Retrieval-augmented generation (RAG) systems address this by fetching relevant documents and injecting them into the context window, but this is a workaround rather than a fundamental solution. The model itself does not learn from the interaction.

The auto-regressive generation process is another bottleneck. Generating a 1,000-token response requires 1,000 sequential forward passes through the model, each producing a single token. This makes inference significantly slower than a single forward pass, and it is the primary reason that serving LLMs at scale requires expensive inference infrastructure. Speculative decoding, where a smaller model drafts tokens that the larger model verifies in parallel, can reduce latency by two to three times, but the fundamental sequential nature remains.

07 The Architecture That Ate AI

The transformer's influence extends far beyond language. Vision transformers (ViTs) treat image patches as tokens and apply self-attention, matching or exceeding convolutional neural networks on image classification benchmarks. The protein-structure predictor AlphaFold 2 uses a variant of the transformer architecture. Audio generation models, code completion systems, and robotic control policies have all adopted transformer backbones. The architecture has become a general-purpose computation engine for any domain that can be expressed as a sequence of tokens.

This generality is both the transformer's greatest strength and its most contested aspect. Critics argue that the field has become dangerously monomorphic, with nearly all research funding flowing toward transformer variants. Alternative architectures like state space models (SSMs), including Mamba and Jamba, promise linear-time inference and constant memory, but have not yet matched transformer quality at scale. Mixture-of-experts architectures, which route tokens to specialized sub-networks, are a transformer modification rather than a replacement.

What is clear is that the transformer, nearly a decade after its introduction, remains the undisputed foundation of practical AI. Every frontier model from every major lab uses some variant of the architecture. The 2017 paper that introduced it has been cited over 150,000 times. Whether the next breakthrough will come from a fundamentally different architecture or from continued refinement of the transformer is the central question of AI research in 2026. For now, attention is still all you need.