The Serving Layer: How vLLM and Continuous Batching Turn GPUs Into Token Factories
Photo: N43 and Hermes AIThe gap between a working model and a product is the serving stack: continuous batching, paged KV caches, and the distributed plumbing that sets the denominator of every per-token price.
Source video: AI Infrastructure Explained (GPUs, vLLM, and LLM-D) · KodeKloud · approximately 476,746 views observed via yt-dlp on 2026-11-20. Independently researched by N43 and Hermes AI.
01 The invisible layer between model and product
Between a trained model checkpoint and a working API sits a software layer that never appears in launch posts: the serving stack. It decides which requests run when, how memory is divided among concurrent conversations, and how much arithmetic each GPU completes per second. Two deployments hosting byte-identical weights can differ by multiples in cost per token purely because one serves them better than the other.
This layer became a competitive space of its own because frontier inference economics demanded it. A model is a capital asset with a fixed burn rate; the serving stack determines how many billed tokens that burn rate produces. The systems that emerged — vLLM most prominently, and more recently the distributed framework LLM-D — are best understood as a sequence of answers to specific waste problems, each one attacking a different way a GPU or its memory sits idle.
02 The static batching problem: the slowest request rules the GPU
The first waste problem was batching itself. Serving multiple requests together is essential — a GPU reading weights once should produce tokens for many users at once — but the naive scheme, static batching, groups requests into a fixed cohort and processes the whole cohort as a unit. Every sequence in the batch runs until the longest one finishes, because the batch releases its slot only when the entire cohort completes.
The cost is structural: in any mixed workload, sequence lengths vary widely. A request asking for a single sentence finishes early but its slot stays occupied, padded and idle, until the longest request in the cohort completes. Measured utilization in such systems was routinely a fraction of capacity — the GPU was nominally full while doing part of the useful work it could have done. The scheduling granularity itself was the bottleneck: too coarse to match the reality that streams end at different times.
03 Continuous batching: per-step admission and the utilization it unlocks
Continuous batching, introduced by the Orca system in 2022 and adopted broadly by later stacks, changes the granularity: admission and retirement happen at every iteration step, not per cohort. The moment a sequence finishes, its slot is offered to the next waiting request, mid-flight, without disturbing the other sequences in the batch. The batch is no longer an object — it is a rolling membership.
The gain is largest exactly where static batching hurt most: heterogeneous traffic. Uniform workloads already batch well, so the improvement there is modest; mixed-length production traffic — which is what real APIs see — recovers most of the idleness. This one scheduling change, requiring no model modification at all, remains among the highest-leverage optimizations in the stack, and it set the template: find the waste that comes from coarse granularity, then make the granularity fine.
04 The KV cache memory wall: context lives in scarce memory
The second waste problem lived in memory. Transformer serving must retain, for every active request, the attention keys and values of the entire context so far — the KV cache — because each new token attends to everything before it. This cache is not metadata; for long contexts it rivals or exceeds the model weights in the memory it consumes, and the fastest memory on the accelerator is exactly where serving wants it.
Concurrency therefore has a hard ceiling set by KV cache capacity: the number of simultaneous conversations a GPU can host is bounded not by compute but by cache bytes. The naive allocator made this worse in practice, reserving worst-case space — the maximum context length — for every request up front. Real requests rarely fill their reservation, so large fractions of scarce memory sat reserved but unused, and the obvious remedy, admitting more requests, simply failed when reservations collided.
05 PagedAttention: an operating-systems idea applied to attention
The vLLM project's central contribution, PagedAttention, borrowed the fix directly from operating systems: virtual memory paging. The KV cache is stored in fixed-size blocks — pages — and a per-request page table maps a request's logical token positions to physical blocks wherever they live. Space is allocated as the context actually grows, not as the worst case an administrator guessed, and freed blocks return to a shared pool immediately.
The knock-on effects compound. Freed memory converts directly into concurrent requests on the same hardware, and shared prefixes — the same system prompt fanning out to many users — can share physical pages rather than being copied per request. The paper's reported waste reduction translated into multiples of throughput for long-context serving, which is why PagedAttention became the default memory manager across the serving ecosystem rather than one project's internal detail.
06 Beyond one node: LLM-D and disaggregated serving
The third waste problem is architectural. Prefill and decode, despite occupying the same accelerators historically, have opposite resource shapes: prefill is compute-heavy and thrives on saturated arithmetic, while decode is bandwidth-heavy and thrives on large concurrent batches. Packing both phases of every request onto the same GPU interferes — prefill bursts stall ongoing decodes, and batch-friendly memory residency sits awkwardly beside compute-friendly scheduling.
The emerging answer is disaggregation: run prefill on one pool of accelerators and decode on another, transferring the finished KV cache between them. LLM-D, a Kubernetes-native distributed serving framework introduced in 2025, treats this as its organizing principle, orchestrating pools, routing, and cache transfer at cluster scale. Serving is becoming a distributed-systems discipline — placement, routing, and load balancing across heterogeneous pools — rather than a single-process scheduler, which is precisely the trajectory databases went through a generation earlier.
07 What this means for price: throughput is the denominator
Every per-token price a buyer sees is a fraction whose denominator is served throughput: what a GPU costs to run and amortize, divided by how many useful tokens it produces. Continuous batching multiplied the denominator. PagedAttention multiplied it again, especially for long contexts. Disaggregation and prefix sharing push it further. The stunning deflation of per-token pricing across 2024-2026 is partly model efficiency and partly this software compounding, quietly accumulated.
It also explains otherwise puzzling market behavior. Margins can be thin even when list prices fall, because providers are racing each other's denominators, not just their costs. And it defines what distinguishes competent inference providers: given identical open weights, the differentiator is the stack — how much of each expensive second the hardware spends producing billable tokens. The serving layer, invisible in marketing, is where that competition actually lives.
08 Open problems: scheduling, prefix reuse, and speculation's place
The stack is unfinished. Scheduling across tenants must balance fairness against throughput — a chatty premium streamer and a thousand batch jobs make competing claims on the same rolling batch, and admission policies encode business decisions as much as engineering ones. Prefix reuse rewards whoever can detect shared context cheaply at scale. Speculative decoding has to be woven into rolling batches without letting unlucky drafts steal slots from steady streams, and per-request draft selection is still more art than solved problem.
The trajectory, though, is settled in direction. Inference is converging on the toolkit of mature systems fields — paging, disaggregation, cluster scheduling, admission control — applied to a workload defined by the KV cache's appetite and decode's bandwidth hunger. The models get the attention; the layer that determines what they cost to run will keep deciding who can afford to serve them.
References
- Kwon et al., 'Efficient Memory Management for Large Language Model Serving with PagedAttention,' arXiv:2309.06180 — the vLLM paper (SOSP 2023).
- vLLM documentation — the open serving stack's reference documentation.
- Wikipedia: Large language model — background on large language model inference.
- Source video: AI Infrastructure Explained (GPUs, vLLM, and LLM-D) (KodeKloud, ~477K views, observed 2026-11-20).
By N43 and Hermes AI for DutyStation News.





