Inside the LLM Training Run: Data, Scale, and Compute

01 The Problem: What a Training Run Actually Is

A frontier training run is best understood as one very large computation with a binary outcome: either it produces a set of weights, or it fails and most of the budget is gone. The inputs are a curated text corpus, a model architecture, and a schedule for feeding tokens to accelerators. The output is not a document or a demo but a parameter file that will spend the rest of its life doing inference. Wikipedia's standard definition of a large language model frames it as an AI model trained on a vast amount of text, and that phrasing undersells the logistics. Corpus assembly, token budgets, checkpointing, and failure recovery turn a training run into an industrial process - closer to a factory campaign than to a laboratory experiment. This article walks through what that factory consumes, stage by stage, and where the money goes.

02 Data Curation: Building the Corpus

Everything starts with the corpus. Teams blend web crawls with licensed archives, code repositories, books, and multilingual collections, then filter aggressively: near-duplicate documents are removed, low-quality pages are scored out, and harmful content is down-weighted. The mixture is not accidental - adjusting the share of code or multilingual text measurably changes what the finished model can do, so labs treat mixture weights as a core recipe secret. Budgets are quoted in tokens, the subword units models actually read. Chart 1 sketches an illustrative token budget for a frontier-scale run in the tens of trillions. Real numbers vary by vendor and are rarely disclosed in full, but the shape is consistent: web-derived text dominates by volume, while code, books, and synthetic instruction data play outsized roles relative to their size because they are dense with exactly the structure training needs.

03 Scaling Laws and the Compute Bill

Scaling laws give the economics their shape. Across many orders of magnitude, loss falls as a smooth function of model size, data volume, and compute, which means a lab can forecast - before spending the money - roughly what a tenfold larger run will buy. That predictability is why budgets exploded: if the curve holds, the rational move is to bid for more accelerators. Public reporting puts landmark runs of the late 2010s in the thousands of dollars, the early 2020s in the hundreds of thousands, and current frontier efforts in the tens of millions, with 2026 runs plausibly crossing one hundred million once hardware, energy, and staffing are counted. Chart 2 shows that trend on a log scale. The figures are order-of-magnitude estimates from public reporting rather than audited costs, but the slope is the story: each era's headline run costs roughly a hundred times the one before it.

04 The Pipeline: Pretrain, Finetune, RLHF

A run is really several runs stitched together. Pretraining consumes the vast majority of compute: the model sees the corpus once or twice, predicting the next token at every step. What emerges is a raw document-completer with broad knowledge and no manners. Post-training then reshapes it. Supervised finetuning on curated demonstrations teaches format and instruction-following; preference optimization, including RLHF-style methods, pushes the model toward answers humans rate highly; further stages add tool use, refusals, and domain skills. These later stages use a tiny fraction of the tokens but a disproportionate share of engineering attention, because alignment failures are more visible than pretraining ones. Checkpoints from every stage are evaluated and sometimes rolled back, and the whole schedule is rehearsed on smaller models first, because a bug discovered mid-run wastes weeks of scarce accelerator time.

05 Evaluation: Measuring What Matters

Knowing whether a run worked is its own discipline. Labs evaluate on public benchmarks, private held-out suites, and structured human preference trials, watching for regressions as closely as gains. Benchmark contamination - test questions leaking into training data - is a standing worry, so teams rotate eval sets and probe for suspiciously memorized answers. Raw scores also diverge from what users actually experience, which is why preference-based evaluations and long-horizon task suites have grown in importance. Evaluation gates the pipeline: a candidate checkpoint that wins on knowledge but loses on safety or instruction-following does not ship. The Stanford CS229 lecture this article accompanies treats evaluation as part of the build rather than an afterthought, and that framing has become standard practice across the industry as runs grow too expensive to redo on a hunch.

06 The Limits: Data Exhaustion and Cost

Two constraints now bind hardest. The first is data: high-quality human-written text is a finite resource, and frontier labs have already absorbed most of what is freely reachable. The responses are synthetic data generated and filtered by existing models, licensing deals over archives that remain closed, and expansion beyond text into other modalities. Each workaround trades one scarcity for another, since synthetic data risks amplifying the flaws of the model that produced it. The second constraint is cost. The slope in Chart 2 cannot continue indefinitely - several orders of magnitude in a few years requires correspondingly enormous revenue. Returns per dollar still exist, but they are hunted through efficiency: distillation into smaller models, better data quality per token, and inference-time compute that substitutes extra reasoning at answering time for raw training scale.

07 What a 2026 Run Looks Like

The 2026 picture is consolidation with exceptions. A handful of labs run frontier-scale campaigns measured in the tens of millions of dollars across huge accelerator fleets, while a second tier runs deliberately smaller efforts that lean on distilled models and open weights. Training is increasingly continuous rather than episodic: refresh runs ingest new data into existing checkpoints instead of starting from scratch, blurring the line between training and deployment. The watch items for the next cycle are data licensing outcomes, whether synthetic pipelines can hold quality at scale, and whether inference-time compute shifts spending from training runs to serving. The factory is not going away - but what it manufactures, and at what margin, is being renegotiated in real time, and the CS229 curriculum now reflects that industrial reality.