AI inference explained: how trained models earn their keep

01 Training is the tuition; serving is the payroll

Training gets the headlines, but a trained model earns nothing until it is run against real requests. Inference is the act of running a trained model to produce outputs: a prompt goes in, the frozen weights are applied, and a completion, classification, or ranking comes out. Every chat reply, autocomplete suggestion, and content-moderation verdict is an inference event, and there are vastly more of them than there were training runs.

The economics of the two phases are shaped differently. Training is closer to a capital expense: an enormous burst of compute, incurred once for a given model. Serving is an operating expense that scales with every user and every token. Once the weights are frozen, the marginal cost of intelligence is measured in tokens per second per dollar of silicon, which is why industry estimates commonly place inference at 80 to 90 percent of cloud AI workload.

This piece walks through the serving stack piece by piece: what actually happens inside a forward pass, why memory rather than raw arithmetic tends to bind, how batching and quantization reshape the cost curve, and where the hardware and energy budgets are pushing the next generation of deployments.

02 One token at a time: the forward pass and the KV cache

Autoregressive language models generate one token at a time. Each new token is conditioned on everything emitted so far, so generation is a loop: embed the sequence, run it through the network's attention and feed-forward layers, predict a distribution over the vocabulary, sample, append, and repeat. A single forward pass is cheap by training standards; the challenge is doing it thousands of times per second, per request, across thousands of requests.

The dominant optimization in that loop is the KV cache. Recomputing attention keys and values for the entire context at every step would multiply work quadratically with sequence length. Instead, serving systems store the keys and values of already-processed tokens, so each step computes attention only for the newest token. The cache trades memory for compute: its size grows with both context length and the number of concurrent requests.

The consequence is that decode-time serving is usually bound by memory bandwidth rather than raw floating-point throughput. Reading cached keys and values and the weight matrices, billions of parameters, from high-bandwidth memory for every generated token is the real bottleneck, and techniques such as paged KV-cache management exist to keep that memory fractionated and fully utilized.

03 Batching: buying throughput with latency

GPUs earn their keep on parallelism. Serving one request at a time leaves most of the matrix units idle, because a single sequence is a thin sliver of work compared with what the hardware can process at once. Batching packs many requests into a single forward pass, letting one sweep of the weights serve dozens of users simultaneously and raising effective GPU throughput dramatically.

The bill for that efficiency arrives as latency. Every request in a batch waits for the whole batch step, and a request that joins a crowded queue inherits the decoding speed of the slowest member. Push batch sizes up and per-request latency climbs even as aggregate throughput keeps improving, a tradeoff every serving product has to price explicitly.

Modern schedulers soften the tradeoff with continuous batching, which admits and retires requests at the granularity of individual decode steps rather than waiting for a whole batch to finish. The curve below is illustrative rather than measured, but its shape, throughput rising steeply then flattening while latency accelerates, is the constraint that shapes capacity planning for every inference deployment.

04 Quantization: smaller numbers, nearly the same answer

Model weights are typically trained in 16-bit floating point, but deployed models rarely need that much precision at inference time. Quantization re-expresses weights, and sometimes activations, as 8-bit or 4-bit integers, cutting memory footprint and energy per token at a small, usually measurable, cost in accuracy. For many production workloads the quality delta is negligible next to the savings.

The deeper effect is on feasibility. A model that does not fit on a given accelerator at 16-bit may fit with room to spare at 4-bit, which changes which hardware can serve it, how much context fits alongside it, and what margin the operator earns. Quantization effectively moves the frontier of what a fixed fleet of chips can host.

The craft is in managing the error. Post-training quantization calibrates integer scales against sample activations, while quantization-aware training teaches the network to expect its reduced precision. As models grow, the difference between a naive and a careful quantization recipe is measured in quality points that users can feel.

05 Token economics

Because generation is sequential, inference cost decomposes into two very different phases. Prefill, processing the prompt, is compute-bound and highly parallel; the whole prompt can be pushed through the network at once. Decode, emitting tokens one at a time, is memory-bound and serial. Long prompts raise admission cost, while long outputs raise the KV-cache pressure and wall-clock time of every subsequent step.

Serving providers price this reality per token, and engineering choices move the price. Prompt caching reuses the prefill work of repeated context, prefix sharing lets many requests amortize a common system message, and speculative decoding lets a small draft model propose tokens the large model verifies in batches. Each technique converts wasted work into margin.

For product builders the unit economics decide viability. A assistant that summarizes a hundred-page document for every query is a different business from one that completes a two-line form, even at identical model quality. Inference is where the abstract benchmark score becomes a line item on an invoice.

06 The hardware landscape

NVIDIA GPUs dominate model serving today, a position built on the CUDA software ecosystem and a decade of optimized kernels as much as on the silicon itself. But the field is no longer uncontested: Google's TPUs, Amazon's Inferentia chips, and a crowd of startup accelerators all make the same bet, that purpose-built silicon can beat general GPUs on tokens per watt per dollar for transformer workloads.

Custom ASICs encode an interesting wager. A general GPU must run any workload well; an inference chip can hard-wire the exact arithmetic of attention and feed-forward layers and shed everything else. If architectures stabilize, fixed-function designs win on efficiency; if architectures keep shifting, flexibility defends the GPU. The 2017-to-present dominance of the transformer has so far rewarded the specialists.

For buyers, the practical consequence is a supply strategy problem. Capacity, contract terms, and software portability now influence which models a product can serve economically, and abstraction layers that keep inference portable across silicon have become a quiet form of insurance.

07 Latency, energy, and the edge

Interactive products live and die on time-to-first-token. Queueing behind other tenants, prefilling a long prompt, and the serial pace of decode all count against a latency budget measured in hundreds of milliseconds. Energy is the sibling constraint: at fleet scale, per-token energy is a design metric, and datacenter power has become a binding input on how much inference capacity can exist at all.

These pressures explain the pull toward the edge. Quantized small models now run on phones and laptops, trading some capability for offline availability, privacy, and the absence of a network round trip. For short, latency-sensitive tasks, a capable on-device model can beat a frontier model behind a cell tower.

The likely equilibrium is hybrid: small models handle the routine locally and delegate the hard queries to datacenter fleets. In every configuration, the principle is the same, the trained model is the asset, and inference is the delivery mechanism that turns it into a service.