How transformers power ChatGPT: the architecture behind modern LLMs

01 The problem before attention

Every sentence is a web of relationships, and the models that came before transformers processed it badly. Recurrent neural networks, the dominant approach into the mid-2010s, read text one token at a time and carried a single compressed summary forward — a hidden state. By the end of a long paragraph, the memory of its opening was squeezed through hundreds of intermediate steps, and the signal decayed. Practitioners called it the vanishing gradient problem; in plain terms, the model gradually forgot how a sentence began.

Worse, reading sequentially left no room for parallelism. Training meant marching through text word by word, so bigger datasets meant proportionally more wall-clock time on hardware that was otherwise scaling beautifully. The field needed an architecture that could look at an entire sequence at once, weigh every word against every other word directly, and still run efficiently on GPUs. In 2017, a Google paper titled "Attention Is All You Need" proposed exactly that, and named the resulting architecture the transformer.

02 Attention, explained simply

The transformer's core idea is self-attention. Text is first split into tokens — word fragments mapped to vectors from an embedding table — and then, at every layer, each token is contextualized against the other tokens in its context window through a parallel multi-head attention mechanism. Concretely, each token emits three vectors: a query (what am I looking for?), a key (what do I contain?), and a value (what do I pass along?). Every query is scored against every key, and each token's output becomes a weighted blend of the values, with important tokens amplified and irrelevant ones diminished. Multiple attention heads do this in parallel, letting the model track different kinds of relationships — syntax, coreference, long-range dependencies — simultaneously.

One detail keeps the whole scheme honest: self-attention alone is permutation-invariant, meaning it sees a bag of tokens with no inherent order. Transformers therefore inject positional information, typically through positional encodings or learned positional embeddings, so the model can distinguish "dog bites man" from "man bites dog." The result is an architecture that processes an entire sequence in parallel, weighs relationships directly rather than through a lossy memory chain, and scales cleanly with hardware — the three properties the pre-transformer era lacked.

03 From transformer to GPT: pretraining on everything

The original transformer had an encoder and a decoder, built for translation. The GPT line kept only the decoder half and asked a brutally simple question: what happens if you train a huge decoder on just predicting the next token across nearly everything written? The answer, demonstrated at increasing scale from 2018 onward, was that next-token prediction is a disguised curriculum. To predict well, a model must absorb grammar, facts, styles of reasoning, and the structure of code — because all of them reduce the prediction error on real text.

This is pretraining, and a large language model is its product: an AI model trained on a vast amount of text for language-generation tasks, capable of generating, summarizing, translating, and analyzing text across many contexts. ChatGPT, Claude, Gemini, Grok, and DeepSeek are all chatbots built on this foundation. What separates a raw pretrained model from a usable assistant is everything layered on top — the instruction tuning and human-feedback training covered below — but the capability ceiling is set by the pretrained base and its architecture.

04 Tokens, context windows, and inference

Using a model is a different process from training one. At inference time, your prompt is tokenized, run through the network, and the model outputs a probability distribution over the next token; sampling from that distribution, appending the result, and repeating is how a whole answer materializes, one token at a time. Nothing is looked up in a database — every word is computed from the interaction of the prompt with billions of learned weights.

The context window is the model's working memory: the maximum number of tokens it can attend to at once, prompt and generated text combined. Its growth is one of the clearest capability curves in the field, shown in Figure 2. Anything inside the window is directly available to attention; anything outside must be re-supplied or retrieved by external tooling. This is why long documents, codebases, and multi-turn conversations stress models differently — and why the jump from a few thousand tokens to hundreds of thousands changed what products could be built, not just how good the answers looked on benchmarks.

05 Scaling laws and why bigger kept helping

The uncomfortable discovery of the GPT era was that scale itself was an algorithm. Empirical scaling laws — formalized by OpenAI researchers in 2020 — showed that model loss falls as a smooth, predictable function of parameters, data, and compute, across many orders of magnitude. Bigger was not just marginally better; it reliably unlocked qualitative new abilities that smaller models simply did not exhibit, a pattern illustrated in Figure 1 by the published parameter counts of the early GPT generations.

Two honest caveats belong next to that chart. First, GPT-4's parameter count has never been disclosed — the published series effectively ends at GPT-3's 175 billion, and later frontier models are widely believed to be sparse mixture-of-experts systems where total parameters no longer equal per-token compute, which is one reason labs stopped advertising the number. Second, scaling is not free: training compute grew by orders of magnitude across the same generations, concentrating frontier development in a handful of organizations. The scaling era's lesson was never "only scale matters," but that scale was a far stronger lever than the field had assumed.

06 RLHF: teaching models to be helpful

A raw pretrained model completes text; it does not answer questions. Ask it a question and it may respond with three more questions, because statistically, that is what internet text looks like. The step that converts a text engine into an assistant is reinforcement learning from human feedback (RLHF). Humans rank pairs of model outputs — which answer is more helpful, accurate, harmless? — and a reward model is trained on those rankings to approximate human preference. The language model is then optimized, by reinforcement learning, to produce outputs the reward model scores highly.

RLHF is why ChatGPT feels like a conversation rather than an autocomplete. It is also a source of the field's characteristic quirks. The model is learning to please an approximation of human judgment, not to be truthful, so sycophancy — agreeing with the user's framing — is a known failure mode. And because the reward model only covers the territory humans sampled, optimization can push behavior into regions where the approximation is wrong. Measured fact: RLHF measurably improves helpfulness ratings. Interpretation: it trades some raw capability and calibration for a personality people want to use, and most practitioners consider that trade worth making.

07 Limits: hallucination, cost, and context

Three limits define the frontier's edge. Hallucination — confident fabrication — follows directly from the objective: a language model is trained to produce plausible next tokens, not verified statements, and nothing in pretraining distinguishes true from merely fluent. Retrieval augmentation and tool use mitigate this by grounding answers in external sources, but the architectural cause remains: the model generates; it does not look things up. Cost is the second limit — every generated token is a full pass through the network, so serving hundreds of millions of users is an economics problem as much as a research one, and per-token price still shapes what products are viable.

The third limit is the context window itself. Attention's cost grows with sequence length, so context is rationed; long conversations push early material out of scope, and what falls outside the window is, to the model, unretrievable. None of these limits are secrets, and all three are active engineering frontiers — longer efficient contexts, cheaper inference, better grounding. The transformer of 2017 solved sequence modeling so well that the field's hardest problems are no longer architectural. They are about truth, cost, and memory — which is to say, about everything that comes after the architecture.