Inside the Transformer: How Large Language Models Actually Work

01 The Core Idea: Next-Token Prediction

At its most fundamental level, a large language model performs a single task: given a sequence of text, predict what token comes next. A token is a fragment of text — a word, a part of a word, or a punctuation mark — drawn from a fixed vocabulary. The model assigns a probability distribution over this vocabulary for each prediction, and the token with the highest probability (or a sampled token from the distribution) is appended to the sequence, which then becomes input for the next prediction.

This deceptively simple objective, when applied to models with billions of parameters trained on trillions of tokens of text, produces behavior that appears to demonstrate understanding, reasoning, and creativity. The model does not actually comprehend language in the way a human does. It learns statistical patterns in its training data so effectively that its output is indistinguishable from comprehension in many contexts — a phenomenon that researchers have described as stochastic parrot behavior, though the field increasingly recognizes that the internal representations learned by these models are richer than simple pattern matching.

02 The Transformer Architecture

The architecture underlying virtually all modern large language models is the Transformer, introduced in the 2017 paper "Attention Is All You Need." The key innovation is the self-attention mechanism, which allows every token in the input sequence to attend to every other token, computing weighted relationships that capture contextual dependencies regardless of distance.

In a self-attention layer, each token is projected into three vector spaces — queries, keys, and values. The attention score between any two tokens is the dot product of one token's query vector and the other's key vector, scaled and normalized through a softmax function. This score determines how much of the other token's value vector contributes to the first token's output representation. Multi-head attention runs this process in parallel across multiple learned subspaces, allowing the model to simultaneously attend to different types of relationships.

03 Training: Pre-training and Fine-Tuning

LLM training proceeds in two major phases. Pre-training consumes the vast majority of compute: the model is trained on a massive corpus of text — web pages, books, code, scientific papers — using the next-token prediction objective. This phase requires thousands of GPUs running for weeks or months, consuming enormous amounts of electricity. The result is a base model that has learned broad patterns of language and knowledge but is not specifically optimized for any task.

Fine-tuning adapts the base model for specific use cases. Supervised fine-tuning trains the model on input-output pairs that demonstrate desired behavior — answering questions, following instructions, writing code. Reinforcement learning from human feedback (RLHF) further refines the model by optimizing responses against a reward model trained on human preferences. This pipeline — pre-train, fine-tune, align — has become the standard recipe, though variations like direct preference optimization are gaining traction as simpler alternatives to RLHF.

04 Tokenization: The Bridge Between Text and Numbers

Before a model can process text, that text must be converted into numbers. Tokenization is the process of splitting text into discrete units — tokens — and mapping each to an integer index. Most modern LLMs use byte-pair encoding or similar subword tokenization schemes that break words into common fragments, allowing the model to handle rare words, misspellings, and multilingual text without an unmanageably large vocabulary.

The choice of tokenizer has significant downstream effects. A model with a larger vocabulary can represent more words as single tokens, reducing sequence length and improving processing efficiency, but at the cost of a larger embedding matrix. Tokenization also affects the model's relationship with characters — an LLM can struggle with tasks like spelling or character-level manipulation because it sees tokens, not individual characters, during training.

05 The Training Data Problem

A large language model is only as good as the data it was trained on. The quality, diversity, and representativeness of the training corpus determine the model's knowledge, its biases, and its failure modes. Biased or inaccurate training data can make an LLM's output less reliable, producing confident-sounding but incorrect responses — the phenomenon known as hallucination.

As frontier models have consumed most of the high-quality text available on the public internet, attention has turned to synthetic data — text generated by other language models — as a way to expand training corpora. This approach raises concerns about model collapse, where successive generations of synthetic-data-trained models degrade in quality. Curating and filtering training data has become a discipline in its own right, with teams dedicated to deduplication, quality scoring, and removal of harmful or low-quality content.

06 Capabilities and Their Limits

Large language models can generate fluent text, summarize documents, translate between languages, write and debug code, answer factual questions, and engage in multi-turn conversations. These capabilities emerge from the training process rather than being explicitly programmed — the model is never taught to translate, but after seeing enough parallel text, it learns the statistical correspondence between languages well enough to produce reasonable translations.

The limitations are equally well-documented. LLMs lack a world model in the human sense — they do not understand physical causality or maintain a consistent model of reality. They can produce contradictory statements within a single response. They struggle with arithmetic and logical reasoning that requires precise multi-step computation. They cannot reliably verify their own output. These limitations stem directly from the architecture: a model trained to predict text patterns will be excellent at producing text patterns, but there is no mechanism ensuring that those patterns correspond to truth.

07 The Economics of Inference

Running a large language model — inference — requires significant computational resources. Each token of output requires a forward pass through the entire network, with the cost scaling roughly linearly with parameter count. For the largest models, serving millions of queries per day requires massive GPU clusters and represents a substantial ongoing expense, in contrast to the one-time cost of training.

This economic reality has driven intense focus on inference optimization: quantization reduces the precision of model weights from 16-bit to 8-bit or even 4-bit with minimal quality loss; speculative decoding uses a smaller model to draft tokens that the larger model verifies; mixture-of-experts architectures route tokens through specialized sub-networks, reducing the effective compute per token. These optimizations are not merely engineering conveniences — they determine whether a model is commercially viable at scale.