How Large Language Models Are Trained

01Start with prediction, not conversation

A base language model does not begin as a helpful chatbot. It begins as a parameterized function that receives a sequence of tokens and predicts the next token. Training supplies enormous numbers of examples where the next token is already present in the text. The model adjusts its weights to make the observed continuation more probable.

This self-supervised setup is powerful because the labels are implicit in the corpus. No one has to hand-write a correct answer for every sentence. But next-token prediction is not the same objective as truth, helpfulness, or harmlessness. Those behaviors require later data and evaluation.

02Text becomes a stream of tokens

Before optimization, documents are cleaned, filtered, deduplicated, and converted into token IDs. Tokenization usually splits text into pieces rather than whole words; common fragments get compact representations while rare strings require more pieces. Padding and batching turn variable-length documents into efficient tensor operations.

Data quality is a training variable. Duplicates can cause memorization; low-quality text can amplify errors; web-scale corpora can contain personal data, copyrighted material, and social bias. The dataset is therefore part of the model’s behavior, not an invisible prelude to “the real” algorithm.

03Pretraining is a repeated compression game

The transformer processes token sequences, produces a probability distribution at each position, and receives a loss when its distribution misses the observed continuation. Gradient descent nudges billions of parameters so that useful statistical regularities become easier to predict. The model is not storing a clean encyclopedia; it is learning distributed patterns that can regenerate, combine, and sometimes distort what it saw.

Scaling laws describe empirical relationships among compute, parameter count, and dataset tokens. Wikipedia records historical cost examples: about $50,000 for the 1.5-billion-parameter GPT-2, about $8 million for the 540-billion-parameter PaLM, and about $11 million for Megatron-Turing NLG 530B. These numbers are reminders that capability is also an infrastructure problem.

04Instruction tuning changes the interface

A pretrained model can complete text without reliably following a user’s intent. Supervised fine-tuning exposes it to examples of prompts and desired responses. The examples teach formatting, task behavior, refusal patterns, and conversational conventions. This stage does not erase pretraining; it steers the same network toward a narrower distribution of outputs.

Evaluation must now test both capability and behavior. A model can become more pleasant while losing factual precision, or become safer on a known test while remaining vulnerable to a new prompt. Alignment is a moving target because users, tasks, and adversaries keep changing.

05RLHF turns preferences into a reward signal

In reinforcement learning from human feedback, people compare candidate answers and indicate which they prefer. A reward model learns to predict those preferences. The language model is then optimized—often with a policy method such as PPO—so its outputs score better under the learned reward while a constraint keeps it from drifting too far from the pretrained policy.

Human feedback is powerful but lossy. Raters may disagree, overlook subtle errors, reward confident style, or represent only a slice of users. The reward model can therefore become a proxy that the policy learns to exploit. Newer direct preference methods can simplify the loop, but no method removes the need for careful data design and independent tests.

06Inference is another training decision

Once deployed, the model generates one token at a time. Temperature, top-k or nucleus sampling, context length, system instructions, retrieval, tool calls, and safety filters all shape the final answer. A frozen set of weights can behave very differently under different serving policies.

The full picture is a pipeline: data curation determines what can be learned; pretraining creates broad predictive competence; fine-tuning shapes interaction; preference optimization changes the reward landscape; inference determines what users actually see. Asking “how was the model trained?” means asking about every stage, not just the number of parameters.