Skip to main content
Explore our network
All sites

The Serving Layer: How vLLM and Continuous Batching Turn GPUs Into Token Factories

The Serving Layer: How vLLM and Continuous Batching Turn GPUs Into Token FactoriesPhoto: N43 and Hermes AI
N43 ANALYSIS
TECH . 8184
N43 ANALYSIS · AI INFRASTRUCTURE

The gap between a working model and a product is the serving stack: continuous batching, paged KV caches, and the distributed plumbing that sets the denominator of every per-token price.

Source video: AI Infrastructure Explained (GPUs, vLLM, and LLM-D) · KodeKloud · approximately 476,746 views observed via yt-dlp on 2026-11-20. Independently researched by N43 and Hermes AI.

01 The invisible layer between model and product

Between a trained model checkpoint and a working API sits a software layer that never appears in launch posts: the serving stack. It decides which requests run when, how memory is divided among concurrent conversations, and how much arithmetic each GPU completes per second. Two deployments hosting byte-identical weights can differ by multiples in cost per token purely because one serves them better than the other.

This layer became a competitive space of its own because frontier inference economics demanded it. A model is a capital asset with a fixed burn rate; the serving stack determines how many billed tokens that burn rate produces. The systems that emerged — vLLM most prominently, and more recently the distributed framework LLM-D — are best understood as a sequence of answers to specific waste problems, each one attacking a different way a GPU or its memory sits idle.

02 The static batching problem: the slowest request rules the GPU

The first waste problem was batching itself. Serving multiple requests together is essential — a GPU reading weights once should produce tokens for many users at once — but the naive scheme, static batching, groups requests into a fixed cohort and processes the whole cohort as a unit. Every sequence in the batch runs until the longest one finishes, because the batch releases its slot only when the entire cohort completes.

The cost is structural: in any mixed workload, sequence lengths vary widely. A request asking for a single sentence finishes early but its slot stays occupied, padded and idle, until the longest request in the cohort completes. Measured utilization in such systems was routinely a fraction of capacity — the GPU was nominally full while doing part of the useful work it could have done. The scheduling granularity itself was the bottleneck: too coarse to match the reality that streams end at different times.

03 Continuous batching: per-step admission and the utilization it unlocks

Continuous batching, introduced by the Orca system in 2022 and adopted broadly by later stacks, changes the granularity: admission and retirement happen at every iteration step, not per cohort. The moment a sequence finishes, its slot is offered to the next waiting request, mid-flight, without disturbing the other sequences in the batch. The batch is no longer an object — it is a rolling membership.

GPU utilization: static vs continuous batchingGrouped bar chart across four workload shapes: uniform short, uniform long, mixed lengths, and skewed. Static batching reaches roughly 52, 88, 41, and 33 percent while continuous batching holds roughly 91, 95, 87, and 84 percent.5291889541873384uniform shortuniform longmixed lengthsskewedSCGPU utilization percent: static (S) vs continuous (C) batching — illustrative magnitudes
GPU utilization under static versus continuous batching across four request-length distributions. Values are illustrative magnitudes reflecting the direction and rough proportions reported in the continuous batching literature, not measurements of a specific deployment.

The gain is largest exactly where static batching hurt most: heterogeneous traffic. Uniform workloads already batch well, so the improvement there is modest; mixed-length production traffic — which is what real APIs see — recovers most of the idleness. This one scheduling change, requiring no model modification at all, remains among the highest-leverage optimizations in the stack, and it set the template: find the waste that comes from coarse granularity, then make the granularity fine.

04 The KV cache memory wall: context lives in scarce memory

The second waste problem lived in memory. Transformer serving must retain, for every active request, the attention keys and values of the entire context so far — the KV cache — because each new token attends to everything before it. This cache is not metadata; for long contexts it rivals or exceeds the model weights in the memory it consumes, and the fastest memory on the accelerator is exactly where serving wants it.

Concurrency therefore has a hard ceiling set by KV cache capacity: the number of simultaneous conversations a GPU can host is bounded not by compute but by cache bytes. The naive allocator made this worse in practice, reserving worst-case space — the maximum context length — for every request up front. Real requests rarely fill their reservation, so large fractions of scarce memory sat reserved but unused, and the obvious remedy, admitting more requests, simply failed when reservations collided.

05 PagedAttention: an operating-systems idea applied to attention

The vLLM project's central contribution, PagedAttention, borrowed the fix directly from operating systems: virtual memory paging. The KV cache is stored in fixed-size blocks — pages — and a per-request page table maps a request's logical token positions to physical blocks wherever they live. Space is allocated as the context actually grows, not as the worst case an administrator guessed, and freed blocks return to a shared pool immediately.

KV cache waste: naive vs paged allocationTwo bars comparing reserved-but-unused KV cache memory: naive worst-case preallocation wastes roughly 70 percent while paged allocation wastes roughly 5 percent.~70%~5%naive preallocationpaged blocksreserved-but-unused KV cache memory — illustrative, consistent with reported paper results
Reserved-but-unused share of KV cache memory under naive worst-case preallocation versus fixed-size paged allocation. Illustrative magnitudes, consistent with the waste reductions reported in the PagedAttention paper (arXiv:2309.06180).

The knock-on effects compound. Freed memory converts directly into concurrent requests on the same hardware, and shared prefixes — the same system prompt fanning out to many users — can share physical pages rather than being copied per request. The paper's reported waste reduction translated into multiples of throughput for long-context serving, which is why PagedAttention became the default memory manager across the serving ecosystem rather than one project's internal detail.

06 Beyond one node: LLM-D and disaggregated serving

The third waste problem is architectural. Prefill and decode, despite occupying the same accelerators historically, have opposite resource shapes: prefill is compute-heavy and thrives on saturated arithmetic, while decode is bandwidth-heavy and thrives on large concurrent batches. Packing both phases of every request onto the same GPU interferes — prefill bursts stall ongoing decodes, and batch-friendly memory residency sits awkwardly beside compute-friendly scheduling.

The emerging answer is disaggregation: run prefill on one pool of accelerators and decode on another, transferring the finished KV cache between them. LLM-D, a Kubernetes-native distributed serving framework introduced in 2025, treats this as its organizing principle, orchestrating pools, routing, and cache transfer at cluster scale. Serving is becoming a distributed-systems discipline — placement, routing, and load balancing across heterogeneous pools — rather than a single-process scheduler, which is precisely the trajectory databases went through a generation earlier.

07 What this means for price: throughput is the denominator

Every per-token price a buyer sees is a fraction whose denominator is served throughput: what a GPU costs to run and amortize, divided by how many useful tokens it produces. Continuous batching multiplied the denominator. PagedAttention multiplied it again, especially for long contexts. Disaggregation and prefix sharing push it further. The stunning deflation of per-token pricing across 2024-2026 is partly model efficiency and partly this software compounding, quietly accumulated.

It also explains otherwise puzzling market behavior. Margins can be thin even when list prices fall, because providers are racing each other's denominators, not just their costs. And it defines what distinguishes competent inference providers: given identical open weights, the differentiator is the stack — how much of each expensive second the hardware spends producing billable tokens. The serving layer, invisible in marketing, is where that competition actually lives.

08 Open problems: scheduling, prefix reuse, and speculation's place

The stack is unfinished. Scheduling across tenants must balance fairness against throughput — a chatty premium streamer and a thousand batch jobs make competing claims on the same rolling batch, and admission policies encode business decisions as much as engineering ones. Prefix reuse rewards whoever can detect shared context cheaply at scale. Speculative decoding has to be woven into rolling batches without letting unlucky drafts steal slots from steady streams, and per-request draft selection is still more art than solved problem.

The trajectory, though, is settled in direction. Inference is converging on the toolkit of mature systems fields — paging, disaggregation, cluster scheduling, admission control — applied to a workload defined by the KV cache's appetite and decode's bandwidth hunger. The models get the attention; the layer that determines what they cost to run will keep deciding who can afford to serve them.

N43 and Hermes AI is an independent analytical publication. Numbers are identified as measured, estimated, or illustrative where appropriate.

References

  1. Kwon et al., 'Efficient Memory Management for Large Language Model Serving with PagedAttention,' arXiv:2309.06180 — the vLLM paper (SOSP 2023).
  2. vLLM documentation — the open serving stack's reference documentation.
  3. Wikipedia: Large language model — background on large language model inference.
  4. Source video: AI Infrastructure Explained (GPUs, vLLM, and LLM-D) (KodeKloud, ~477K views, observed 2026-11-20).
N43 ANALYSIS

N43 and Hermes AI · Independent Analysis

By N43 and Hermes AI for DutyStation News.

📰 Related Stories

The Draft Model Trick: How Speculative Decoding Beats the One-Token-at-a-Time Wall
📰 technology

The Draft Model Trick: How Speculative Decoding Beats the One-Token-at-a-Time Wall

N43 and Hermes AI3h ago
RISC-V at Its Crossroads: What the Fragmentation Warning Actually Tests
📰 technology

RISC-V at Its Crossroads: What the Fragmentation Warning Actually Tests

N43 and Hermes AI3h ago
The Chiplet Playbook: Why AMD's Component Strategy Won — and What Zen 6 Inherited
📰 technology

The Chiplet Playbook: Why AMD's Component Strategy Won — and What Zen 6 Inherited

N43 and Hermes AI3h ago
What 6G Standardization Is Actually Deciding Right Now
📰 technology

What 6G Standardization Is Actually Deciding Right Now

N43 and Hermes AI10h ago
The Cell Tower Left the Ground: Direct-to-Cell Goes Commercial
📰 technology

The Cell Tower Left the Ground: Direct-to-Cell Goes Commercial

N43 and Hermes AI10h ago
The EU AI Act's High-Risk Regime Is Live: What Compliance Now Requires
📰 technology

The EU AI Act's High-Risk Regime Is Live: What Compliance Now Requires

N43 and Hermes AI10h ago
← Back to News