How Frontier LLMs Are Actually Trained and Served

01Why serving, not training, is the hard half

A frontier model begins life as an inert artifact: a file of hundreds of billions of floating-point numbers, the learned weights, plus a recipe describing how they fit together. The Wikipedia definition is blunt about the origin. A large language model is an AI model trained on a vast amount of text for language-generation tasks, and everything a chatbot later says is computed from those frozen numbers. Training produces the weights; it does not produce answers.

Serving is the discipline of turning that file into a live conversation, and it is the half of deployment outsiders never see. Training is a batch problem, run once over weeks, with nobody watching the clock. Serving is an interactive problem, run millions of times a day, under latency budgets set by human impatience. Every single request must stream the same enormous weights out of accelerator memory before the first word appears, and that one physical fact shapes the entire engineering stack above it.

The asymmetry explains a peculiar industry pattern: leading labs treat serving infrastructure as a secret on par with the model itself, and improvements that never make a changelog - a better scheduler, a tighter memory kernel - can quietly change how fast every user's answers arrive. A recent long-form conversation between interviewer Dwarkesh Patel and inference engineer Reiner Pope takes exactly this shift as its premise: the frontier of AI engineering has moved, substantially, from how models are made to how they are run. This article unpacks the machinery behind that claim, section by section, and marks where established mechanism ends and interpretation begins.

02From weights to answers: prefill and decode

Every request moves through two phases with completely different shapes. Prefill reads the prompt: the model runs a forward pass over all input tokens at once, in parallel, computing the internal state needed to begin answering. Because the work arrives as one large, dense batch of matrix multiplications, prefill keeps the accelerator's arithmetic units busy and is, per token, the cheap phase of the request.

Decode is the phase users actually watch. The model emits one token at a time, and every token requires another complete pass over the weights. The arithmetic in each step is small next to the memory traffic of reading hundreds of gigabytes of parameters, so decode is memory-bound: the chip spends most of its time moving weights, not computing. This is why output tokens arrive more slowly than input tokens are processed, and why every modern chatbot streams its answer rather than posting it whole at the end.

The split matters because the phases compete for the same hardware in different ways. Prefill wants big batches to stay efficient; decode wants company, so the traffic of streaming weights is shared across many simultaneous conversations. Production schedulers juggle the two, sometimes splitting them onto separate pools of machines entirely. Where a request's wall-clock time actually goes is the subject of the first chart below, drawn for the long-prompt, short-answer case that makes the split starkest.

03The KV cache and the economics of batching

The bridge between the phases is the KV cache: a block of working memory holding the keys and values that each layer computed for tokens already processed. With it, generating token five hundred means attending over cached state; without it, the model would reprocess the entire conversation from scratch for every token. The cache is why a chat stays fast as it grows long, and it is also the reason serving is a memory problem before it is a compute problem.

The cache grows linearly with context length and with the number of simultaneous users, and the growth is steeper than most people expect. The second chart below sketches it for an illustrative 70B-class reference model at fp16 precision: roughly 2.5 GB of cache at an 8K context, about 10 GB at 32K, and about 40 GB at 128K for a single conversation. Numbers in that range explain, immediately, why long-context features arrive slowly and why schedulers fight so hard over memory.

Batching is the countermove. Rather than dedicating an accelerator to one conversation, the scheduler interleaves dozens or hundreds, so the memory traffic of streaming weights is amortized across many parallel decodes. Continuous batching refines the idea by letting new requests join mid-flight instead of waiting for a batch to drain. The trade is a classic one: fuller batches raise utilization but add jitter to everyone's latency, and caches held open for many users at once consume exactly the memory that longer contexts would need.

04Accelerators and the datacenter footprint of a chatbot

None of this runs on general-purpose CPUs. Frontier serving happens on specialized accelerators - Google's Tensor Processing Units, NVIDIA GPUs, and custom silicon from other large operators - clustered in datacenters and stitched together with high-bandwidth interconnects. The lineage is visible in the Wikipedia records for both chips: processors originally built for graphics or for narrow matrix workloads became the substrate for machine learning because training and inference reduce to the same primitive, multiplying very large matrices very quickly.

The binding constraint is memory capacity and memory bandwidth rather than raw arithmetic. A single accelerator cannot hold a frontier model's weights plus the caches of hundreds of users, so both are sharded across many chips, and a single generated token may require coordination across that cluster. When it does, the interconnect - the wiring between chips - can become the real ceiling on speed, which is why datacenter topology shows up in how responsive a chatbot feels from your pocket.

It also explains the quiet geography of a conversation. Your prompt may be processed in one facility and decoded in another; routing logic prefers to keep a long conversation pinned to machines that already hold its cache, because moving that state is slow and the transfer itself competes with paying traffic. Latency, in other words, is a property of a distributed system, not of the model file alone.

05The quality-latency-cost triangle

Serving teams balance three pressures that pull against each other: the quality of answers, the latency of responses, and the compute and memory burned per request. The levers mostly control how much work happens per answer: sampling settings, output-length limits, how aggressively long contexts are compressed, and where a product sits on the spectrum from fast-and-loose to slow-and-careful generation.

Speculative decoding shows how the triangle is worked. A small draft model proposes several tokens; the large model verifies all of them in a single pass and keeps the ones it would have produced anyway. Verification parallelizes, so answers speed up without changing the output distribution - the same quality, sooner - at the price of serving two models and scheduling two interlocking workloads. Variants of the trick are now standard across the industry.

The triangle has no free vertex. Shorter answers cut memory pressure and waiting but can thin out quality; longer contexts improve grounding but inflate caches; denser batching raises throughput but adds tail latency for whoever shares the batch. Serving engineering is the craft of choosing, per product and per request, which corner gives way first.

06What the serving stack means for model releases

Once the stack is visible, release notes read differently. Architectures are chosen with serving in mind: grouped-query attention exists in large part because it shrinks the KV cache; mixture-of-experts designs trade memory footprint for arithmetic; long-context launches are gated on whether caches can be managed without collapsing latency. A model that decodes slowly is a worse product even when it scores better on benchmarks, and every major lab knows it.

Many announcements are infrastructure announcements in disguise. When output speed jumps between versions, the model may have changed - or the scheduler, the kernels, or the hardware behind it may have. Smaller distilled variants exist because the flagship checkpoint is impractical to serve for every tier of traffic. Even API limits - context windows, maximum outputs, requests in flight - are the serving stack explaining its own physics to developers in the only language an interface can speak.

The practical lesson for anyone building on these systems: evaluate the deployed pair, not the artifact. The same weights behind different serving stacks can feel like different products. Streaming behavior, how long conversations degrade, and the wait before the first token are measurable properties of model plus infrastructure, and they deserve the same scrutiny as benchmark scores when you choose what to ship.

07Limits and open problems

The stack is impressive and unfinished. Decode remains memory-bound; caches still grow linearly with context; and the batching techniques that keep accelerators busy were designed for chat-shaped traffic. Agentic workloads - bursts of requests, long-running sessions, tool calls between generations - stress those assumptions in ways the current generation of schedulers handles imperfectly.

Research is attacking the cache directly: compressing or summarizing old context, paging cold caches to lower tiers of memory, and sharing common prefixes such as system prompts across users so they are stored once rather than thousands of times. Each approach trades some exactness or engineering complexity for capacity; none has made the underlying pressure disappear.

The deepest open question is directional. Serving today optimizes for conversation; the frontier is moving toward agents that work for minutes or hours between human check-ins. Whether the reigning architecture - weights streamed per token, state held per conversation - survives that shift is precisely the sort of question serving engineers are now best placed to answer, and that, more than any single benchmark, is why the serving stack has become the industry's quiet center of gravity.