Ask an inference server to answer one question and you get one GPU doing something odd: running a matrix multiplication for a single sequence. GPUs are throughput machines — they earn their keep when thousands of rows flow through each matmul. During interactive LLM serving, though, requests arrive one at a time, produce wildly different numbers of output tokens, and finish at unpredictable moments. How you group them onto the GPU is the difference between a server that costs ten dollars a day and one that costs two hundred.
This post walks through continuous batching — the scheduling strategy behind every serious open-source inference engine, including vLLM and NVIDIA’s TensorRT-LLM — why the naive alternative leaves most of your silicon idle, and which scheduling problems appear once you adopt it. By the end you’ll know what the engine is doing on every forward pass, why a long prompt can stall every other user, and which knobs control the trade-offs.
Why static batching wastes the GPU
The obvious way to batch LLM work is the way batch inference has always worked: collect N requests, run them all together, wait until every one is done, then start the next group. This is static batching, and for LLMs it fails for a structural reason: output length is not known in advance. One request might need four tokens (“Yes.”), another four hundred (a full code review). The batch only completes when its longest member finishes.
Consider a batch of eight requests with generation lengths of 20, 45, 210, 33, 8, 67, 400, and 12 tokens. After roughly twenty decode steps, half the sequences are finished — but their slots in the batch keep executing. Each decode step runs the full batch tensor through the model, so those finished sequences are pure waste: the GPU spends FLOPs and memory bandwidth generating nothing. Measured naively, static batches commonly run with well under half the batch doing useful work in the later steps, and the scheduler can’t even top the batch up, because the group is frozen until everything drains.
Padding makes it worse. Batching requires every sequence in the step to share a tensor shape, so short prompts get padded up to the longest one. You pay compute — and attention over padding tokens — for rows that contain no information.
Iteration-level scheduling: the core idea
Continuous batching (also called in-flight batching or dynamic batching with iteration-level scheduling inverts the unit of scheduling. Instead of forming a batch of requests, the scheduler forms a batch of forward passes. At every iteration it asks: which sequences still have work to do right now? Those go into the batch tensor. A sequence that emits its end-of-generation token leaves immediately, and the next queued request is admitted into the vacated slot on the very next step.
There is no “group” to finish, so there are no stragglers holding slots hostage. The batch membership churns constantly while the batch tensor stays full. This is where LLM serving gets its characteristically high utilization: decode steps are memory-bandwidth-bound, and a full batch amortizes the weight-reading cost of each step across as many live sequences as possible.
One thing that makes this cheap to implement is that the decode step for every sequence is the same shape of work: one token in, one token out, attention over that sequence’s KV cache. New sequences join mid-flight without disturbing the others, because each row of the batch is independent — attention never crosses sequence boundaries.
The problem continuous batching creates: long prefills
Admitting a new request means running its prefill — processing the entire prompt in one shot to populate the KV cache before the first token can come out. While other requests are decoding (each step is tiny), a prefill for a 30,000-token prompt is an enormous compute-heavy iteration. Everything else in the batch waits for it: their next token is delayed by the full prefill duration. Under mixed traffic — one user pasting a large document, dozens of users chatting — that one big prefill becomes an inter-token latency spike for everyone in the engine.
Chunked prefill fixes this by splitting a prompt into fixed-size token chunks and scheduling them across several iterations. Each chunk extends the request’s KV cache, and subsequent chunks attend to everything processed before them, so the mathematical result is identical to a single-pass prefill — the first token just appears after the last chunk lands. The scheduler can then mix one prefill chunk with the decode tokens of all active requests in the same forward pass. This pairing works because prefill chunks are compute-bound while decode is memory-bound: the chunk saturates the GPU’s compute capacity, and the decode rows ride along at near-zero marginal cost.
In current vLLM releases chunked prefill is enabled by default whenever possible; on older versions it was opt-in. The remaining knob is the token budget per iteration:
vllm serve meta-llama/Llama-3.1-8B-Instruct --max-num-batched-tokens 8192The piggybacking trick above — pairing chunked prefills with decodes to fill pipeline bubbles — is studied in depth in the SARATHI paper, if you want the arithmetic behind why mixing the two phases works so well on modern accelerators.
Memory pressure, preemption, and fairness
A continuous scheduler admits requests as long as there is room, and the binding constraint is almost never tensor capacity — it’s KV cache memory. Every admitted sequence holds KV blocks proportional to its context length. When the cache fills, the scheduler has two options: refuse new requests (waiting queue grows) or preempt — evict a running sequence, release its KV blocks, and either recompute the sequence from its prompt later or swap its cache to CPU memory. vLLM’s default preemption mode is recompute, which trades wasted FLOPs for zero host-device copying and is usually the cheaper option.
Preemption shows up in logs as a warning, and if you see it repeatedly, your batch size ambitions exceed your memory budget:
WARNING scheduler.py Sequence group 0 is preempted by
PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.There’s also a fairness problem the basic scheduler ignores. Admission is roughly first-come-first-served, so a burst of long-context requests can starve short ones, and two clients with identical request rates can see very different latencies depending on when their prompts happened to land. Production systems layer explicit policies on top — token budgets per iteration shared across clients, priority queues, or fairness-aware batch formation — but the baseline engines make no promise here, so measure per-client latency, not just aggregate throughput.
The knobs that matter in production
Continuous batching itself is not a tuning decision — in vLLM it is always on, with no flag to disable it. What you tune is the shape of the batch:
--max-num-seqs— the maximum number of sequences running concurrently. Higher values raise throughput until KV cache pressure triggers preemption; then latency degrades sharply. Find the knee empirically.--max-num-batched-tokens— the token budget per forward pass, shared between prefill chunks and decode tokens. Smaller budgets bound the worst-case iteration time (better inter-token latency); larger budgets favor throughput and faster time-to-first-token for big prompts.- KV cache size (
gpu_memory_utilization) — determines how many sequences fit before preemption starts. This is usually the real ceiling on concurrency.
TensorRT-LLM exposes the same scheduling model under the name in-flight batching, and NVIDIA’s benchmarks attribute a meaningful share of its throughput gains over naive serving to it, so the concept transfers across engines even when the flags do not.
From the client side, continuous batching means the engine’s throughput holds up under concurrency — but your application still experiences queueing. Streamed responses hide the waiting better than blocking calls, because the first token of a long prefill request arrives while the engine keeps serving everyone else:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
stream = client.chat.completions.create(
model="meta-llama/Llama-3.1-8B-Instruct",
messages=[{"role": "user", "content": "Summarize the attached document."}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)Wrapping up
Continuous batching is the scheduling model that makes LLM serving economical: schedule forward passes, not request groups; refill slots the moment they empty; and use chunked prefill to keep long prompts from stalling everyone else. The remaining levers — concurrency caps, per-iteration token budgets, and KV cache sizing — trade throughput against latency, and the right setting depends on whether your traffic is chat-shaped, batch-shaped, or a mix. Start from the defaults, watch preemption warnings, and load-test with realistic output-length distributions, because that variance is exactly what the scheduler is designed to absorb.