Large language models explained: how tokens, training and tuning actually work

01What a large language model actually is

A large language model is an AI model trained on a vast amount of text for natural language processing tasks, especially language generation. That definition, from the plainest corner of the technical literature, is worth unpacking, because every word in it does real work. "Trained" means the model's behavior was not programmed by hand — it was squeezed out of statistical patterns in the data. "Vast" means internet-scale: trillions of words drawn from books, code, articles, and web pages. "Generation" is the key capability: these systems can produce, summarize, translate, and analyze text in many contexts, and they are the basis for many modern chatbots, such as ChatGPT, Claude, Gemini, Grok, and DeepSeek.

Mechanically, an LLM is a neural network — most commonly a transformer — with billions of adjustable numerical knobs called parameters. During training, the model repeatedly tries to predict the next word in a passage of real text, compares its guess against reality, and nudges every parameter to make the error infinitesimally smaller. Repeat that loop trillions of times and the network ends up encoding a startling amount of grammar, fact, style, and reasoning structure — not stored as sentences in a database, but smeared across the parameters themselves.

02Tokens and the limits of context

Models do not read words; they read tokens. A tokenizer chops text into fragments — often a few characters each — and maps every fragment to an ID number the network can process. "Unbelievable" might split into three tokens; a space plus a short word is usually one. Common words get efficient single tokens, rare words shatter into pieces, which is why models sometimes trip over spelling puzzles and invented words.

Context is the second hard limit. A model's context window is the maximum number of tokens it can consider at once — everything it will ever "know" about your conversation has to fit inside it. Modern systems have pushed windows from a few thousand tokens to hundreds of thousands or more, enough to hold entire codebases or novels. But the window is a budget, not a library: attention degrades with distance, older content can get crowded out, and cost scales with every token included. The practical skill of using an LLM well is largely the skill of packing the right information into a limited token budget.

03Pretraining on internet-scale text

Pretraining is the expensive part, and the word "expensive" deserves its own paragraph. A frontier training run involves thousands of GPUs or accelerators working for weeks or months, consuming electricity and capital in comparable measure. The model passes over a curated corpus of trillions of tokens — filtered for quality, deduplicated, and increasingly supplemented with licensed and synthetic data — and simply learns to predict the next token, over and over, across effectively the entire internet's worth of text.

What emerges from pretraining is a base model: a superb next-token predictor with no particular desire to be helpful. It will happily complete a question with three more questions, or continue a medical passage with confident nonsense, because that is what the raw text distribution suggests. The base model knows how language behaves; it does not yet know how an assistant should behave. That gap is what the next stage exists to close.

04Fine-tuning and alignment

Fine-tuning starts from the pretrained base and reshapes its behavior. The classic recipe is supervised fine-tuning: humans write tens of thousands of exemplary question-and-answer exchanges, and the model is trained to reproduce their tone and structure. On top of that, labs apply reinforcement learning from human feedback — the model generates many candidate answers, humans rank them, and a reward model learns to predict those rankings, then nudges the policy toward answers people prefer.

Alignment, in the broad sense, is the set of techniques for making the tuned model safe and honest: refusal training for harmful requests, red-teaming to surface failure modes, constitutional-style methods that give the model explicit principles to critique itself against. The honest summary is that alignment works reasonably well in practice but is not a solved problem. Fine-tuning moves probability mass onto helpful behaviors; it cannot add knowledge the base model lacks, and it cannot guarantee the model will never produce something its trainers would have ranked poorly. The tuned model is the same network as the base — steered, not replaced.

05Hallucination and why models guess

Hallucination is the field's most public relations problem, and its explanation is almost embarrassingly simple. A language model is a probability distribution over next tokens. When you ask a question whose answer exists in the training data, the distribution's peak sits on the correct answer. When you ask about something the model never learned — or only partially learned — the distribution still has a peak. The model has no concept of silence; it will always place its bet somewhere, and when the evidence is thin, the bet is a guess dressed in fluent grammar.

Worse, tuning for helpfulness can amplify the problem: a model trained to always produce a complete, confident-sounding answer learns that hedging scores worse with raters than guessing cleanly. The mitigations are now standard practice — retrieval augmentation that bolts real documents into the context window, tool use that outsources arithmetic and search, calibration training that rewards saying "I don't know." They reduce hallucination's frequency without eliminating it, because its root cause is architectural: a next-token predictor with no internal flag for "this territory is unmapped."

06Where LLMs go next

The frontier is moving in three directions at once. First, scale continues: newer generations add parameters, data, and compute, and measurable capability keeps arriving with each increase — the pattern that makes claims about "scaling is over" look premature year after year. Second, agency: the models are being wrapped in scaffolding that lets them call tools, browse, execute code, and chain steps together, turning a text predictor into the reasoning core of a working system. Third, multimodality and efficiency: today's leading models read images, audio, and video natively, and a parallel track of research — quantization, distillation, on-device inference — pushes meaningful capability onto phones, where it runs without a datacenter.

For the non-specialist, the durable takeaways from a thirteen-minute explainer are these: an LLM is a statistical engine that predicts text, not a database that retrieves it; its limits — context, knowledge cutoffs, hallucination — follow directly from that design; and its abilities, which can seem inexplicable, follow from it just as directly. Understanding the mechanism is what separates a tool user from a tool believer. The Gradient Descent's walkthrough is a solid thirteen minutes to spend on the former.