The LLM Stack Explained: From Training Data to Token Generation

01 The Stack Mentality

Andrej Karpathy, a founding member of OpenAI and former director of AI at Tesla, has spent years explaining that a large language model is not a single technology but a stack of interdependent layers. His one-hour introductory talk, viewed nearly four million times, maps the full pipeline from raw training data to the moment a user receives a generated response. Understanding the stack—rather than treating the model as a monolith—is what separates informed analysis from mythology.

Each layer in the stack solves a different problem. The data layer determines what the model knows. Tokenization determines how text is represented. Pretraining teaches the model to predict the next token. Fine-tuning adapts the model to follow instructions. Alignment makes the model safe and helpful. Inference turns a prompt into a response. A weakness at any layer propagates upward, and a strength at one layer can partially compensate for a weakness at another—but only partially.

02 Data: The Foundation

Everything a language model knows comes from its training data. The data layer is not glamorous, but it is the most consequential decision in the entire stack. A model trained on scientific papers will know physics; a model trained on Reddit will know memes. Most frontier models are trained on a mixture: web crawl data, books, code repositories, scientific papers, and curated conversation datasets. The composition, quality, and deduplication of this mixture determine the model's knowledge, its biases, and its blind spots.

The scale of training data has grown enormously. Early language models were trained on billions of tokens; modern frontier models consume trillions. But more data is not automatically better. Low-quality data—machine-generated text, duplicated content, spam—can degrade model performance. The most sophisticated training pipelines now include aggressive filtering, deduplication, and quality classification to ensure that the model learns from high-quality text rather than noise.

03 Tokenization: The Bridge Between Text and Math

Language models do not process text; they process numbers. Tokenization is the bridge. A tokenizer splits text into chunks called tokens—typically shorter than a word, longer than a character—and assigns each chunk a unique integer ID. The word "tokenization" itself might become the tokens "token" and "ization," each mapped to a number the model can process.

Tokenization is not trivially correct. Different tokenizers split text differently, and the choice affects the model's performance. A tokenizer that splits common words into many sub-tokens forces the model to learn relationships across token boundaries. A tokenizer that handles code well but handles non-Latin scripts poorly creates a model that writes better Python than Mandarin. The tokenizer also determines the context window: a model with a 128,000-token context window can process approximately 100,000 English words, but fewer characters of Chinese, because Chinese characters tokenize less efficiently.

04 Pretraining: Learning to Predict

Pretraining is the most expensive and most important phase. The model is shown trillions of tokens and asked to predict the next token given the preceding context. Every time it gets the prediction wrong, the parameters are adjusted through gradient descent to make the correct prediction more likely next time. After trillions of these corrections, the model has learned grammar, facts, reasoning patterns, style, and a vast amount of world knowledge.

The pretraining objective—next-token prediction—is remarkably simple, but it forces the model to learn complex internal representations. To predict the next word accurately in a sentence about quantum mechanics, the model must understand quantum mechanics. To predict the next word in a Python function, it must understand Python. The model is never explicitly taught these things; it learns them because accurate prediction requires understanding.

Pretraining is also where the scaling laws manifest. Research has shown that model performance improves predictably with three factors: more parameters, more data, and more compute. The relationships are power laws—each factor contributes a measurable, logarithmic improvement. This is why frontier models have grown from hundreds of millions of parameters to over a trillion: the scaling laws say bigger is better, and so far they have been correct.

05 Fine-Tuning: From Prediction to Instruction

A pretrained model can predict the next token, but it cannot follow instructions. If you ask a pretrained model "What is the capital of France?", it might respond with another question rather than an answer, because its training objective was to continue text, not to respond helpfully. Fine-tuning closes this gap by training the model on examples of instructions and correct responses.

Instruction fine-tuning uses datasets of prompt-response pairs: "Summarize this article" paired with a good summary, "Translate this sentence" paired with a correct translation, "Write Python code for this task" paired with working code. The model learns not just the content of the responses but the format—when asked a question, it should answer directly, not continue with more questions. This phase uses far less data than pretraining—millions of examples rather than trillions of tokens—but it has an outsized effect on usability.

Domain-specific fine-tuning adapts the model further. A model fine-tuned on medical literature will perform better on medical questions. A model fine-tuned on legal documents will produce more accurate legal analysis. The base model's broad knowledge is preserved, but the fine-tuning sharpens performance in the target domain. This is why companies build on open-source base models and fine-tune for their specific use cases rather than training from scratch.

06 Alignment: Making Models Safe

Alignment is the process of making models behave in ways that are helpful, honest, and harmless. The dominant technique is reinforcement learning from human feedback (RLHF): human raters compare model outputs, and the model is trained to prefer the outputs humans prefer. More recent methods like direct preference optimization (DPO) simplify the process by removing the reinforcement learning step, but the principle is the same—shape the model's behavior through human judgment.

Alignment is where the most visible controversies live. A model that refuses to answer benign questions because its safety filters are too aggressive is over-aligned. A model that provides instructions for harmful activities is under-aligned. The balance is difficult because human preferences are inconsistent: users want models that are helpful but not dangerous, confident but not reckless, opinionated but not biased. The alignment layer encodes these trade-offs, and no configuration satisfies everyone.

07 Inference: The Moment of Truth

Inference is the top of the stack—the layer users actually interact with. When you type a prompt, the model tokenizes it, looks up the embedding for each token, runs the stacked transformer layers to build a contextualized representation, and outputs a probability distribution over the vocabulary for the next token. The token is sampled, appended to the context, and the process repeats until a special stop token is generated or a length limit is reached.

The inference layer is also where the engineering challenges are most visible. Running a trillion-parameter model requires significant compute: either a powerful GPU for local inference or a data center for cloud-based models. The latency between prompt and response is determined by the model's size, the prompt's length, and the hardware's speed. Optimizations like quantization (reducing parameter precision), speculative decoding (using a smaller model to draft tokens that the larger model verifies), and caching (reusing computation for repeated prefixes) have made inference faster and cheaper, but the fundamental cost of running large models remains substantial.