Next-Token Machines: How LLMs Turn Context Into Language

01 Text becomes a sequence

An LLM begins by splitting text into tokens: words, subwords, punctuation, or other pieces. Tokenization makes language manageable for a numerical model, but it also creates quirks. A name, a rare word, or a programming symbol may be represented by several pieces. The model sees positions and token IDs, not human concepts in the form we experience them.

02 Training is prediction at scale

During pretraining, the model repeatedly sees context and learns to predict a following token. Gradient-based optimization adjusts a large set of parameters so the predictions become statistically better across its training mixture. The objective is deceptively simple; the data pipeline, compute, architecture, and evaluation choices determine what capabilities and biases emerge.

03 Attention is contextual bookkeeping

Transformer attention lets each position weigh information from other positions. In broad terms, layers build representations that combine local syntax with longer-range relationships. This is not a lookup table of definitions. It is a learned computation whose internal features can support translation, code completion, summarization, and reasoning-like behavior without providing a human-readable guarantee of why a sentence is correct.

04 Fluent is not the same as true

The model’s training signal rewards plausible continuations, not direct contact with the world. Inaccurate data, underspecified prompts, distribution shifts, and decoding can yield confident errors—often called hallucinations. Retrieval, tools, citations, constrained outputs, and human review can improve reliability, but each adds its own failure modes.

05 Capability has a systems boundary

What users call “the model” is usually a stack: tokenizer, neural network, prompt, context window, decoder, safety policy, tools, and application logic. A benchmark score cannot fully predict behavior in a messy workflow. Latency, cost, privacy, data freshness, and the ability to recover from errors are part of intelligence-in-practice.

06 The useful mental model

Think of an LLM as a high-dimensional conditional generator. It compresses patterns from training into parameters, transforms the current context, and emits one token at a time. That framing avoids two opposite mistakes: treating it as a mind with guaranteed beliefs, or dismissing it as mere autocomplete. The engineering question is not whether it “understands” in the abstract, but where its learned behavior is dependable—and how the surrounding system catches the rest.