How Large Language Models Actually Work

Chart: Approximate parameter counts for notable LLMs. GPT-4 estimate is based on published analyses. Source: manufacturer announcements and published research.

01 The Transformer Revolution

The transformer architecture, introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al., fundamentally changed how machines process language. Before transformers, recurrent neural networks like LSTMs processed text sequentially, one token at a time, which made parallel training difficult and limited the context a model could effectively use. Transformers replaced recurrence with self-attention, a mechanism that lets every token in a sequence directly attend to every other token, regardless of distance.

This architectural shift had two profound consequences. First, it enabled massive parallelization during training, since attention computations across a sequence could be distributed across thousands of GPU cores simultaneously. Second, it allowed models to capture long-range dependencies in text that recurrent networks simply could not reach. The combination of parallelizable training and long-range context created the conditions for scale to become the primary lever of progress.

The core insight was deceptively simple: instead of processing words one at a time through a hidden state, compute attention scores between all pairs of tokens in a sequence simultaneously. These scores, normalized through softmax, determine how much each token contributes to the representation of every other token. Multi-head attention extends this by running multiple attention computations in parallel, each learning different relational patterns.

02 Tokens and Embeddings

Before a language model can process text, it must convert human language into numbers. This happens through tokenization, a process that splits text into discrete units called tokens. Modern tokenizers like Byte-Pair Encoding (BPE) and SentencePiece break text into subword units that balance vocabulary size with coverage. A token might represent an entire word, a common suffix, or a single character for rare words.

Each token is mapped to a high-dimensional vector called an embedding. These vectors start as random numbers but are learned during training to capture semantic relationships. The famous property of embedding spaces is that arithmetic operations on these vectors can encode meaning: the vector for "king" minus "queen" approximately equals "man" minus "woman." This geometric structure emerges from training, not from explicit programming.

Positional encodings are added to token embeddings so the model knows the order of tokens in a sequence. The original transformer used sinusoidal positional encodings, but modern variants have introduced learned positional embeddings, rotary position embeddings (RoPE), and ALiBi, each addressing the challenge of helping models understand position without sacrificing the attention mechanism's distance-agnostic nature.

03 The Training Pipeline

Modern LLM training happens in stages. The first is pre-training, where the model learns to predict the next token in a sequence across trillions of tokens of internet text, books, and code. This phase consumes the vast majority of compute resources and can take months on thousands of GPUs. The model learns grammar, factual knowledge, reasoning patterns, and world knowledge from this self-supervised objective.

After pre-training, the model can generate text but has no notion of helpfulness, safety, or following instructions. Supervised fine-tuning (SFT) addresses this by training on curated examples of instruction-response pairs. The model learns to format outputs as answers rather than continuations. This stage uses far less data and compute than pre-training but dramatically changes the model's output distribution.

Reinforcement Learning from Human Feedback (RLHF) is the final stage. Human raters compare multiple model outputs for the same prompt and rank them. A separate reward model is trained on these preferences, and the main model is optimized against this reward signal using Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO). RLHF aligns the model with human preferences for helpfulness, honesty, and safety, though it can also reduce output diversity and introduce subtle biases.

Chart: Approximate training compute estimates. Values are based on published analyses and manufacturer disclosures. GPT-4 compute is estimated.

04 Emergent Abilities

One of the most debated phenomena in LLM research is the emergence of capabilities that were not explicitly trained for. As models scale up in parameter count, training data, and compute, they develop abilities like arithmetic, translation, code generation, and multi-step reasoning that smaller models lack. Researchers have observed that these capabilities often appear suddenly rather than gradually, leading to the term "emergent abilities."

The mechanism behind emergence remains contested. Some researchers argue that emergence is partly an artifact of evaluation metrics: if a task requires getting many components right simultaneously, a model that is 80% correct on each component still fails the task until it reaches a threshold where all components align. Others argue that scale genuinely unlocks qualitatively different reasoning strategies. The truth likely involves both factors, and disentangling them remains an active research area.

What is clear is that the relationship between scale and capability is not linear. Doubling model size does not double performance on most benchmarks. Instead, different capabilities improve at different rates, and some capabilities plateau while others continue to improve. This non-uniform scaling makes it difficult to predict what a model at a given scale will be able to do, which has practical implications for deployment decisions.

05 Limitations and Hallucinations

Large language models do not have beliefs, knowledge, or understanding in the human sense. They generate text by sampling from a probability distribution over the next token, conditioned on the preceding context. This statistical process can produce outputs that are factually correct, eloquently argued, and genuinely useful, but it can also produce confident, fluent, and completely wrong statements, a phenomenon called hallucination.

Hallucination is not a bug that can be fully eliminated. It is a structural property of how LLMs work: the model has no ground-truth verification mechanism. Techniques like retrieval-augmented generation (RAG), where the model is given relevant documents to ground its responses, can reduce hallucination rates significantly. But the model still generates text token by token, and there is no guarantee that it will faithfully represent the retrieved information.

Other limitations include context window size, which constrains how much text the model can consider at once; training data cutoffs, which mean the model has no knowledge of events after its training data was collected; and susceptibility to prompt injection, where adversarial inputs can override the model's instructions. These limitations are not merely academic; they have real consequences when LLMs are deployed in medical, legal, or financial contexts where accuracy is critical.

06 The Road Ahead

The LLM landscape in 2026 is characterized by three parallel trends. First, the largest proprietary models from OpenAI, Anthropic, and Google continue to push scale boundaries, with training runs consuming orders of magnitude more compute than GPT-4. Second, open-source models from Meta, Mistral, and DeepSeek have narrowed the gap with proprietary models, making capable LLMs available to anyone with a consumer GPU. Third, efficiency improvements in architecture, training, and inference are making it possible to run increasingly capable models on smaller hardware.

Multimodal models that process text, images, audio, and video together represent the next frontier. Models like GPT-4o and Gemini can reason across modalities in a single architecture, enabling applications that were impossible with text-only models. The integration of multimodal perception with agentic capabilities, where models can use tools, browse the web, and execute code, points toward systems that act rather than merely respond.

The open question is whether scale alone will continue to yield qualitative improvements, or whether architectural innovation will be needed to overcome the diminishing returns that some researchers observe. Mixture-of-experts architectures, sparse attention mechanisms, and new training objectives are all being explored. What seems certain is that the transformer, seven years after its introduction, remains the dominant architecture, and the race to build more capable language models is accelerating rather than slowing down.