How Large Language Models Actually Work: The Architecture Behind AI Text Generation

01 The Tokenization Bottleneck

Before a language model can process any text, that text must be broken into discrete units called tokens. Tokenization is not merely splitting on whitespace; it is a learned process that maps subword fragments to integer identifiers. The dominant algorithms, Byte-Pair Encoding (BPE) and WordPiece, build vocabulary tables by iteratively merging the most frequent character pairs in a training corpus. A typical modern tokenizer operates with a vocabulary of 32,000 to 256,000 tokens, and the choice of vocabulary directly affects how efficiently the model can represent different languages and technical domains.

The tokenization step creates a fundamental information bottleneck. Every concept the model encounters must be expressed through this fixed vocabulary, which means that rare words are decomposed into multiple subword fragments. The word "uncharacteristically" might be split into "un", "character", "istic", "ally" — four tokens instead of one. This fragmentation affects both the model's processing cost and its ability to maintain semantic coherence over long passages. Models with larger vocabularies reduce this fragmentation but pay a penalty in embedding matrix size, creating a design trade-off that has shaped every major LLM architecture.

02 The Transformer Architecture

The transformer, introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al., replaced recurrent architectures with a design built entirely on attention mechanisms. The key innovation was the self-attention operation, which allows every position in a sequence to directly attend to every other position, eliminating the sequential bottleneck that limited RNNs and LSTMs. A transformer layer consists of a multi-head self-attention sublayer followed by a position-wise feed-forward network, each wrapped in residual connections and layer normalization.

Modern LLMs stack dozens of these transformer layers — GPT-4 uses a decoder-only architecture with an estimated 96 to 120 layers. Each layer refines the representation produced by the previous one, building increasingly abstract features from the raw token embeddings. The depth of the stack determines the model's capacity for compositional reasoning, while the width (the hidden dimension) controls the richness of individual representations. The balance between depth and width is one of the most consequential architectural decisions in model design.

03 Attention: The Core Mechanism

Self-attention computes a weighted sum of value vectors for every position, where the weights are determined by the similarity between query and key vectors. The operation is expressed as the scaled dot-product attention formula: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V. The scaling factor sqrt(d_k) prevents the dot products from growing too large in magnitude, which would push the softmax function into regions with extremely small gradients and slow training.

Multi-head attention runs several attention operations in parallel, each with its own learned projection matrices, and concatenates the results. This allows the model to attend to different aspects of the input simultaneously — one head might focus on syntactic relationships, another on semantic similarity, and a third on positional patterns. The number of attention heads has grown from 8 in the original transformer to 96 or more in the largest contemporary models. The computational cost of attention is O(n^2 * d) in sequence length, which is why context window expansion has been such a hard engineering problem.

04 Training: Pre-training and Fine-Tuning

The training of a large language model proceeds in stages. The first stage, pre-training, exposes the model to trillions of tokens of text scraped from the web, books, and code repositories. The objective is next-token prediction: given a sequence of tokens, predict the probability distribution over the next token. This seemingly simple task forces the model to learn grammar, factual knowledge, reasoning patterns, and stylistic conventions. The loss function is cross-entropy, and training typically requires thousands of GPU-years of compute.

After pre-training, the model undergoes fine-tuning to align its behavior with human preferences. Supervised fine-tuning (SFT) trains the model on high-quality demonstration data, while reinforcement learning from human feedback (RLHF) uses a reward model trained on human preference rankings to optimize the policy via proximal policy optimization (PPO) or similar algorithms. The transition from a base model to a chat assistant involves a substantial behavioral shift, and the quality of this alignment step is often what distinguishes a good model from a great one.

05 The Context Window Problem

The quadratic cost of self-attention means that doubling the context window quadruples the memory and compute required for a single forward pass. Early transformer models operated with context windows of 512 or 2,048 tokens. By 2026, several models support windows of 128,000 to 2,000,000 tokens, but this expansion has required significant architectural innovations. Sparse attention patterns, where each token attends only to a subset of positions, reduce the O(n^2) cost to O(n * sqrt(n)) or better. Ring attention distributes the computation across multiple devices, andFlashAttention optimizes the memory access patterns to avoid materializing the full attention matrix.

The context window is not just a technical parameter — it determines what tasks the model can perform. A 4,096-token window is sufficient for short question-answer interactions but cannot process a research paper or a codebase. The expansion to million-token windows has opened new application categories, including document-level analysis, multi-file code generation, and long-form reasoning, but it has also increased inference costs dramatically, creating tension between capability and economics.

06 Inference: From Probabilities to Words

At inference time, the model generates text one token at a time. Each forward pass produces a probability distribution over the entire vocabulary, from which the next token is selected. The simplest strategy is greedy decoding, which always picks the highest-probability token, but this produces repetitive and predictable text. Temperature sampling introduces randomness by dividing the logits by a temperature parameter before applying softmax — higher temperatures produce more diverse but less coherent output, while lower temperatures are more focused but can become deterministic.

Top-p (nucleus) sampling, used by most production systems, dynamically selects the smallest set of tokens whose cumulative probability exceeds a threshold p, typically 0.9 or 0.95. This adapts to the shape of the distribution: when the model is confident, only a few tokens are considered; when it is uncertain, more candidates are included. The choice of decoding strategy has a measurable impact on output quality, and in practice it is as important as the model itself in determining the user experience.

07 Scaling Laws and Their Limits

The Kaplan scaling laws, published by DeepMind in 2020, described a power-law relationship between training compute, model size, and dataset size, with loss decreasing predictably as any of these factors increases. The Chinchilla refinement in 2022 showed that the original laws had over-emphasized model size relative to data: for compute-optimal training, the dataset should scale roughly proportionally with model parameters. This insight reshaped training strategies across the industry, leading to much larger data collection efforts and more careful data curation.

By 2026, there are growing signs that the scaling laws are flattening. Simply adding more parameters and data yields diminishing returns for general capabilities. The frontier has shifted toward architectural efficiency (mixture-of-experts models that activate only a subset of parameters per token), synthetic data generation, and test-time compute scaling, where the model is given more time to "think" before producing an answer. Whether these approaches can sustain the pace of improvement that pure scaling delivered from 2020 to 2024 remains an open question.