The Mathematics of Language: How LLMs Turn Text Into Understanding

01 From Words to Numbers: The Embedding Problem

Before a language model can process text, it must convert words into numbers. This is not a trivial encoding like assigning each word an ID. Instead, every word, subword, or token is mapped to a high-dimensional vector, a list of hundreds or thousands of floating-point numbers. These vectors are called embeddings, and they encode meaning geometrically: words that appear in similar contexts end up with vectors that point in similar directions in a high-dimensional space.

The idea traces back to the word2vec algorithm introduced by Tomas Mikolov and colleagues at Google in 2013. They discovered that training a shallow neural network to predict surrounding words produced vector representations where semantic relationships emerged as vector arithmetic. The classic example: the vector for "king" minus the vector for "man" plus the vector for "woman" lands near the vector for "queen." This was not programmed into the model. It emerged from the statistics of language.

Modern LLMs use far more sophisticated embedding schemes. Tokens are not whole words but fragments, allowing the model to handle any text including misspellings and novel compounds. The embedding dimension has grown from a few hundred in early models to over 12,000 in the largest current systems. Each dimension captures some latent feature of meaning, though these features are rarely interpretable in human terms. The embedding space is a mathematical landscape where distance encodes semantic similarity and direction encodes analogy.

02 Attention: The Mechanism That Changed Everything

The transformer architecture, introduced by Vaswani et al. in 2017, replaced recurrent networks with a mechanism called self-attention. Instead of processing words one at a time in sequence, the transformer looks at all words simultaneously and computes how much each word should attend to every other word. This parallel processing is why transformers can be trained efficiently on massive GPU clusters.

Self-attention works through three matrices derived from each token's embedding: queries, keys, and values. For each token, the model computes a dot product between its query vector and every other token's key vector, producing attention scores. These scores are normalized through a softmax function into weights that sum to one. The output for each token is then a weighted sum of all value vectors, where the weights are the attention scores. In practice, this is computed in matrix form for efficiency, and the operation is repeated multiple times in parallel through multi-head attention, allowing the model to attend to different aspects of the input simultaneously.

The number of attention heads has grown from 8 in the original transformer to over 100 in frontier models. Each head learns to attend to different patterns: some focus on local syntax, others on long-range dependencies, and others on specific semantic relationships. The combined output of all heads is projected back into the embedding dimension and passed through a feed-forward network, which applies a nonlinear transformation to each position independently.

03 The Training Objective: Predicting the Next Token

Despite their complexity, LLMs are trained on a deceptively simple objective: predict the next token. Given a sequence of tokens, the model outputs a probability distribution over the entire vocabulary, typically 50,000 to 200,000 tokens. The training signal is cross-entropy loss, which measures how much probability the model assigned to the actual next token compared to all alternatives. Minimizing this loss across billions of training examples is what teaches the model grammar, facts, reasoning patterns, and style.

The training data for frontier models is measured in trillions of tokens, drawn from web pages, books, code repositories, and synthetic data. The model sees each example only a few times during training. What it learns is not memorization but statistical regularity: which words tend to follow which, how arguments are structured, what patterns of reasoning appear in mathematical proofs, and how to maintain coherence across long passages.

The scaling laws, formalized by Kaplan et al. in 2020 and refined by Hoffmann et al. in 2022, describe a power-law relationship between training compute, model size, and dataset size. The key finding: model performance improves predictably as you increase compute, parameters, and data in the right proportions. This empirical regularity is what drove the industry to invest billions in larger models and more training data.

04 Inside the Transformer: Layers, Heads, and Residuals

A modern LLM is a stack of transformer layers, typically 32 to 120, each containing a multi-head attention block and a feed-forward network. Between and within these blocks, residual connections add the input of each sublayer to its output, creating paths through which gradients can flow directly during training. Layer normalization stabilizes the distribution of activations. The cumulative effect of passing through dozens of these layers is that the model builds increasingly abstract representations of the input.

Early layers tend to capture syntactic features: part-of-speech tags, sentence boundaries, and local grammatical relationships. Middle layers encode semantic information: entity properties, factual associations, and topic clustering. Later layers are where task-specific behavior emerges: next-token prediction, instruction following, and reasoning chains. This layered abstraction is not designed but discovered through training, and researchers study it through techniques like probe classifiers and activation patching.

The feed-forward network in each layer is where most of the model's parameters reside. It consists of two linear transformations with a nonlinear activation function between them. The hidden dimension is typically four times the embedding dimension, meaning a model with 8,000-dimensional embeddings has feed-forward layers with 32,000 hidden units. This expansion and contraction allows the model to store and retrieve factual knowledge, acting as a kind of associative memory.

05 The Gap Between Prediction and Understanding

LLMs are sometimes described as stochastic parrots, systems that merely repeat statistical patterns without genuine understanding. The critique has merit: the model's objective is prediction, not comprehension. It does not have beliefs, intentions, or a world model in the human sense. Yet the distinction between pattern matching and understanding is less clear than it first appears.

When an LLM solves a mathematical problem, it must identify the relevant technique, apply it step by step, and produce a correct answer. Whether this constitutes understanding or sophisticated pattern matching is a philosophical question, not a purely empirical one. What is empirically clear is that larger models trained on more data develop capabilities that smaller models lack, including multi-step reasoning, code generation, and cross-domain analogical thinking. These emergent abilities appear suddenly as models cross certain scale thresholds, though the sharpness of emergence is debated.

The practical implication is that LLMs can be wrong in ways that are difficult to predict. They can produce fluent, confident text that contains factual errors, logical mistakes, or fabricated citations. This phenomenon, called hallucination, is a direct consequence of the training objective: the model is optimized for plausible continuation, not factual accuracy. Mitigations like retrieval-augmented generation and reinforcement learning from human feedback reduce but do not eliminate the problem.

06 The Economics of Inference: Why Running LLMs Is Expensive

Training a frontier LLM costs tens to hundreds of millions of dollars in compute alone. But inference, the process of generating text from a trained model, is where ongoing costs accumulate. Each token generated requires a forward pass through every layer, and the key-value cache that stores attention information grows linearly with context length. For a model with 100 billion parameters processing a 100,000-token context, a single forward pass involves trillions of floating-point operations.

The industry has responded with a suite of optimization techniques. Quantization reduces the precision of model weights from 16-bit to 8-bit or even 4-bit, cutting memory usage and increasing throughput with minimal quality loss. Speculative decoding uses a small draft model to propose tokens that the large model verifies in parallel, sometimes doubling generation speed. Mixture-of-experts architectures route each token to a subset of the model's parameters, so only a fraction of the network is active for any given prediction.

These optimizations matter because the economics of LLM deployment depend on cost per token. At current prices, running a frontier model costs roughly $0.01 to $0.06 per 1,000 output tokens, depending on the provider and model. For applications that generate millions of tokens per day, this adds up quickly. The trajectory of inference cost reduction, driven by better hardware and algorithmic improvements, will determine how broadly LLMs are deployed across consumer and enterprise applications.

07 What Comes Next: Beyond Next-Token Prediction

The next-token prediction objective has taken the field remarkably far, but researchers are exploring alternatives. Test-time compute, where models spend more inference time on harder problems, has shown that the same model can produce better answers when allowed to search, verify, and backtrack. Systems that combine LLMs with external tools, retrieval, and structured reasoning loops are extending capabilities beyond what raw scale alone would suggest.

The frontier of LLM research in 2026 is defined by several tensions: between scale and efficiency, between generality and specialization, between open and closed models. Open-weight models from organizations like DeepSeek and Meta have narrowed the capability gap with proprietary systems, while proprietary models from OpenAI, Anthropic, and Google continue to push the upper bound. The mathematical foundations remain the same: embeddings, attention, and next-token prediction. What is changing is how these components are combined, scaled, and deployed.