What It Actually Takes to Train an LLM in 2026

01 From Scratch Is a Supply Chain, Not a Model

Say from scratch in 2026 and most listeners picture an architecture: attention layers stacked in some novel arrangement. The teams that have actually done it describe something closer to a heavy-industry project. Before a single gradient step, a from-scratch build requires a data plant that ingests web-scale crawls and refines them into trillions of filtered, deduplicated, quality-scored tokens; a tokenizer trained on a corpus sample and frozen early; a distributed training stack that keeps thousands of accelerators productive through weeks of runs; and an evaluation harness that can say, daily, whether the thing is learning. The model definition is one component among at least five, and rarely the one that decides success.

This is the framing the Stanford CS229 lecture on building large language models takes, and its endurance is telling: roughly 3,090,000 views on a lecture whose first third is about data pipelines and training objectives rather than transformer internals. Audiences keep arriving for the architecture lesson and staying for the accounting lesson, because the accounting is what separates a paper from a product. A syllabus that ends at the attention equation produces engineers who can modify a model; a syllabus that starts at the corpus produces engineers who can build one.

The lens matters for reading the rest of this piece. We are not going to explain how attention works. We are going to follow the bill: what the data costs, what the compute costs, how the token budget gets chosen, and where the modern practice deliberately departs from the textbook guidance written a decade of scaling ago.

02 Data: Curation at Trillions of Tokens

The raw input to a serious 2026 pretraining run is a filtered slice of the public web measured in tens of trillions of tokens, of which the run actually consumes a curated subset. The pipeline looks less like downloading and more like refining: extract text from markup, drop boilerplate and boilerplate-adjacent spam, deduplicate at the document and paragraph level, score what remains with classifiers trained to imitate quality judgments, then mix the survivors with curated high-value sources, code, books, papers, encyclopedic text, in proportions that are themselves an active research problem. Every stage is a place where silent regressions hide, which is why serious teams treat data versions with the same rigor as code versions.

The published numbers give the scale. GPT-3 trained on roughly 0.3 trillion tokens in 2020, which was considered enormous at the time. Chinchilla in 2022 established a new reference point near 1.4 trillion. Llama 3 in 2024 consumed on the order of 15 trillion tokens, a fifty-fold increase over three years, most of it not because the web got bigger but because curation got better: more aggressive filtering meant the same raw crawl could yield far more usable tokens without degrading the model. Data engineering, not data acquisition, is the binding craft.

It also quietly sets the ceiling of the whole enterprise. Tokens are the one input whose quality cannot be bought directly, only manufactured, and the gap between a good mixture and a mediocre one at identical compute is large enough to decide races. When a frontier lab says its advantage is data, it means years of pipeline iteration: classifier generations, mixture experiments, contamination audits. That asset does not show up in parameter counts, and it does not transfer with a model checkpoint.

03 Compute: Learning to Read the Exponents

Training budgets are quoted in FLOP, floating-point operations, and the numbers only make sense on a logarithmic axis. GPT-2 in 2019 needed roughly 1e21 operations, a billion trillion, which felt obscene at the time. GPT-3 jumped to about 3e23 a year later. Llama 3 405B in 2024 ran near 4e25, and the largest 2025 and 2026 runs sit around 1e26. That is five orders of magnitude in roughly six years, which in ordinary engineering terms means the field rebuilt its factory every eighteen months and then doubled the factory again.

Reading the chart honestly requires the log caveat stated out loud: the bars encode exponents, so the visual gap between GPT-3 and Llama 3 405B is two and a bit orders of magnitude, a factor of somewhat over a hundred, and the gap between GPT-2 and a frontier run is a factor of about a hundred thousand. Cost follows the same curve. At 2026 accelerator prices, an exponent is a line item; 1e26 FLOP is not a bigger budget than 1e25, it is a different funding category, the difference between a corporate skunkworks and national infrastructure.

The practical consequence for anyone building from scratch is that compute planning is the first design decision, not the last. The exponent you can afford determines the parameter count you can train, which determines the token budget, which determines the data plant size. Teams discover this in the wrong order and end up with a model shaped by ambition rather than by arithmetic, and the arithmetic always wins by mid-run, when the checkpoint schedule makes the shortfall irreversible.

04 Chinchilla, and Why Every Serious Run Breaks It

In 2022, the Chinchilla result gave the field its first honest budget rule: for a model of a given parameter count, trained on a fixed compute budget, loss is minimized at roughly twenty tokens of training data per parameter. A 70-billion-parameter model, by that logic, wants about 1.4 trillion tokens, and Chinchilla itself trained on close to exactly that, which is why the paper reset everyone's intuitions. For two years, twenty-to-one was the ratio quoted in every planning doc, the closest thing scaling had to a speed limit.

Then the frontier deliberately started breaking it, and the reason is economics rather than science. Chinchilla optimizes training cost per unit of quality; it says nothing about who pays after training. A deployed model runs billions of times, so inference cost dominates its lifetime bill, and the cheapest lifetime configuration is a larger model squeezed harder during training. Llama 3 405B is the canonical case: trained on roughly 15 trillion tokens, nearly twice the Chinchilla prescription for its size, an overtraining strategy that made the resulting checkpoint dramatically cheaper to serve at a given quality level.

This is the quiet lesson of the token chart: the modern frontier is not where compute-optimal training says it should be, because the objective changed from training efficiency to lifetime efficiency. Distillation deepens the same logic, since a large overtrained teacher can compress into small students that no pretraining ratio would have produced. Anyone citing twenty-to-one in 2026 as a design target is quoting a training-time rule to answer a deployment-time question, and the mismatch explains a surprising share of bad budget plans.

05 Tokenization: The Unglamorous Middle

Between the data plant and the accelerator cluster sits the component nobody photographs: the tokenizer, the fixed dictionary that maps text into the units the model actually reads. It is trained early, frozen, and almost never revisited, which makes its decisions permanent infrastructure. Vocabulary size sets a three-way trade: a larger dictionary means fewer tokens per document, so every downstream context window and every inference bill stretches further, but the embedding tables grow and rare languages and code fragments fragment into awkward pieces. Teams that treat tokenization as a detail discover the cost in production, where serving economics are decided per token.

The failure modes are as instructive as the economics. Tokenizers trained on web text under-represent arithmetic strings, non-Latin scripts, and domain formats, and a model cannot easily exceed the granularity of its vocabulary: if every number from a sensor log or every identifier from a codebase shatters into five tokens each, those domains start with a handicap no amount of downstream fine-tuning fully removes. Multilingual costs differ by double-digit percentages purely as a function of tokenization efficiency, a tax assigned before training begins.

The from-scratch lesson is sequencing. Because the tokenizer is frozen while everything else evolves, it has to be built for where the product is going, not where the demo is: the languages you intend to serve, the code you intend to read, the structured text you intend to ingest. Retrofitting a vocabulary means retraining from zero, which at 2026 budgets is not a patch but a rebuild. It is the clearest example of a small component whose decisions echo through an exponent-sized bill.

06 The Systems Layer: Keeping Ten Thousand Chips Busy

At 1e26 FLOP, the limiting resource is not raw hardware but utilization: a frontier run keeps tens of thousands of accelerators in a single synchronized computation for weeks, and every percentage point of idle silicon is a line item paid at full price. The engineering answer is a stack of parallelisms applied simultaneously, data parallelism across replicas, tensor parallelism splitting individual matrix multiplies, pipeline parallelism staggering layers, each added to buy utilization at the price of communication overhead. Tuning that mix is closer to operations research than to deep learning, and it routinely determines whether a run finishes in the fiscal year that funded it.

Failure is the other permanent resident. At cluster scale, hardware faults stop being exceptional; a weeks-long run assumes that memory chips, interconnects, and whole nodes will die mid-flight, so checkpoint-restart is not a nicety but the core scheduling problem. The published engineering write-ups converge on the same grim arithmetic: plan for regular interruptions, verify checkpoints by restoring from them, and treat silent data corruption, hardware that returns wrong answers without complaining, as a real enough threat that teams compute known answers on production hardware to hunt it.

This is why the systems layer is now a moat. Architectures publish in PDFs within days; the operational knowledge of how to hold utilization above sixty percent through a two-month run on flaky hardware travels in headcount, not papers. A from-scratch team can copy a model design accurately and still fail at this layer, and from the outside the failure looks mysterious: the loss curve simply stalls, or the run ends, and the interesting part never gets written up.

07 What a From-Scratch Run Buys in 2026, and the Outlook

The honest closing question is whether anyone should do this at all. Open-weight releases have pushed competence in every non-frontier tier toward zero marginal cost, and for most products, fine-tuning or distilling an existing checkpoint dominates a from-scratch run on every axis except control. What a pretraining run actually buys is the items that cannot be licensed: the data plant and its mixture judgments, the tokenizer tuned to your domains, the base model whose biases and capabilities you chose rather than inherited, and a research staff that understands the system at every layer. Organizations that need those items are rare, and mostly they are already doing it.

The cost curve still points down in specific places. Data curation and synthetic generation keep raising the yield of each token; distillation keeps moving capability into smaller, cheaper boxes; and inference-optimized training, the Chinchilla-breaking logic of section four, keeps shifting spend from training to serving. The frontier keeps its own schedule, though: each order of magnitude of compute has so far bought a new tier of capability, and the teams at 1e26 are not spending that way to replicate 1e25.

For the near term, watch three numbers. The token yields teams extract per raw crawl, which measure the data craft; the utilization frontier teams report on ever-larger clusters, which measure the systems craft; and the ratio of training spend to lifetime inference spend in new budget plans, which measures how thoroughly the post-Chinchilla economics have been absorbed. A lecture recording from Stanford CS229 laid this stack out years ago for free, and the field, whatever its secrets, still teaches its own accounting in public.