Inside the Transformer: The Architecture Behind Every Modern LLM

01From recurrence to attention

Every large language model you have used since roughly 2018 is built on the same skeleton: the transformer, an architecture introduced in the 2017 paper “Attention Is All You Need.” Before that, the dominant tools for language were recurrent neural networks (RNNs) and LSTMs, which read a sentence one token at a time and carried a running “memory” forward. That design had a built-in flaw: information from early words faded as the sequence grew longer, and the strict left-to-right order meant step 100 could not begin until step 99 was finished.

The transformer replaced that running memory with attention, a mechanism that lets every position in a sequence look directly at every other position and decide, with learned weights, which ones matter. Long-range dependencies no longer degrade with distance; they are one direct connection away. This single change is why the transformer displaced RNNs almost completely within a few years, first in machine translation and then everywhere else.

02Tokens and embeddings: text as numbers

A transformer does not see words; it sees tokens, sub-word fragments produced by a fixed vocabulary and a compression scheme such as byte-pair encoding. Common words may be a single token, while rarer words split into pieces. Each token ID is mapped, through a learned lookup table, to an embedding: a vector of a few thousand numbers that positions the token in a high-dimensional space where similar meanings sit near each other.

Because attention itself has no built-in sense of order, the model adds a positional signal to each embedding so that “dog bites man” and “man bites dog” do not collapse into the same input. The exact scheme has evolved, from the original sinusoidal positions to learned and rotary encodings, but the job is the same: inject sequence order into a fundamentally order-blind computation.

03Inside the block: self-attention and the MLP

The core of the architecture is a stack of identical blocks. Each block contains two sub-layers. The first is multi-head self-attention: every token emits three vectors, a query, a key, and a value. Each query is compared against all keys, the similarities are normalized with a softmax into attention weights, and each token receives a weighted mixture of values. Multiple heads run this process in parallel, letting different heads specialize, one tracking syntax, another coreference, another long-range topic structure.

The second sub-layer is a position-wise feed-forward network (MLP), applied to each token independently, which is where a large share of the parameters and much of the model’s factual storage is believed to live. Residual connections and normalization wrap both sub-layers so gradients can flow through dozens or hundreds of stacked blocks without vanishing. GPT-style decoder-only models use one direction of attention (each token sees only its past), while encoder models such as BERT let tokens see in both directions.

04Parallelism: why transformers scaled

The decisive practical advantage is not architectural elegance but hardware fit. An RNN’s recurrence is inherently sequential, so adding compute does not shorten training. Self-attention over a whole sequence, by contrast, is a set of large matrix multiplications that can be executed for every position at once. That parallelism maps directly onto thousands of GPU cores, which is what made training on trillions of tokens economically feasible.

The same property changed the scaling playbook. Once training time stopped depending on sequence order, the reliable path to better models became bigger: more parameters, more data, more compute. As the charts below show, that recipe produced a roughly ten-thousand-fold rise in landmark model sizes between 2017 and 2024, and an even steeper climb in training compute.

05What GPT-style pretraining actually does

A model like GPT is trained on one deceptively simple objective: given the tokens so far, predict the next one. There are no labels in the human-annotation sense; the supervision is the text itself, which is why pretraining scales to internet-sized corpora. Gradient descent nudges billions of parameters so that the probability assigned to each actually-occurring next token goes up. Repeated over trillions of tokens, this forces the model to internalize grammar, facts, style, and many reasoning-like patterns just to keep its predictions competitive.

The result of pretraining is a base model: a next-token distribution, not an assistant. The chatbot behavior users see comes afterward, from fine-tuning on curated instruction data and from reinforcement learning on human or automated preference feedback. Those later stages shape how the model responds; they add relatively little new knowledge compared with pretraining.

06Scaling: parameters and compute

Empirical scaling laws formalized the bigger-is-better trend: loss falls predictably as a power law in parameters, data, and compute. Later work, notably the Chinchilla analysis, showed that many earlier models were under-trained, that parameter count and training tokens should grow together, and that simply inflating parameters without enough data wastes compute.

07The limits: context windows and hallucination

Attention’s all-pairs comparison carries a cost that grows with the square of sequence length, which is why the context window, the maximum number of tokens a model can consider at once, has historically been limited and why long-context variants need engineering tricks such as sliding windows, sparsity, or compressed state. A model cannot attend to what falls outside its window, and even inside it, effective use of very long context is weaker than raw capacity suggests.

The deeper limit is epistemic: a language model generates the most statistically plausible continuation, not a verified one. When the plausible continuation requires a fact the model absorbed loosely or not at all, it produces fluent, confident text that is wrong, the failure mode known as hallucination. Retrieval augmentation, tool use, and citation-based grounding reduce the exposure, but none changes the core mechanism; plausible is not the same as true, and every user of these systems inherits that gap.