Every team that serves language models eventually hits the same wall: the GPU has plenty of memory and the model is small enough, yet throughput is embarrassingly low and tokens trickle out one slow batch at a time. In most cases the culprit is not the model or the hardware — it is batching inherited from training. Training batches are static by design: gather N examples, run them through the network, done. Generation does not work that way, and forcing request-level batching onto token-level work wastes most of the compute you are paying for.
This post walks through continuous batching — also called iteration-level scheduling or in-flight batching — the scheduling mechanism behind every modern serving runtime, from vLLM to SGLang and TensorRT-LLM. You will learn why static batching starves the GPU, how per-iteration scheduling fixes it, which knobs actually matter in production, and where the approach still falls short.
Why static batching wastes the GPU
Consider a classic static batch: four requests arrive, and the server runs them together through the model until every sequence has emitted its final token. Request A might want a five-token answer (“yes, exactly.”), while request D is generating a 2,000-token code review. The batch is only as fast as its slowest member. For the entire duration of request D’s generation, requests A, B, and C are finished — but their slots in the batch stay occupied, and any new request that arrives in the meantime waits in a queue doing absolutely nothing.
The waste compounds because generation is memory-bandwidth-bound. At a batch size of one, decoding a token with an 8B-parameter model in bf16 means reading roughly 16 GB of weights from HBM to produce a single token — the arithmetic itself is trivial. The only way to amortize that weight-read cost is to decode many sequences in parallel so each pass over the weights yields a token for every one of them. A static batch that spends most of its life half-empty is paying full bandwidth prices for a fraction of the possible output.
Padding makes it worse. Batched matrix operations need uniform tensor shapes, so shorter sequences get padded up to the longest member — compute spent on tokens nobody wants. And because a batch cannot start until it is full (or a timeout fires), the first token of every request arrives late: time-to-first-token (TTFT) suffers for everyone queued behind a partially filled batch.
Iteration-level scheduling: the Orca insight
The fix, introduced at scale by the Orca paper at OSDI 2022, is to move the scheduling decision from the request level down to the iteration level. An iteration is one forward pass of the model. Instead of locking in a batch of requests for their entire lifetime, the scheduler re-evaluates the batch composition on every single forward pass: sequences that finished their generation are evicted immediately, and waiting requests are admitted into the vacated slots at the next iteration.
The scheduling loop is conceptually tiny:
# Simplified illustration of iteration-level scheduling.
while True:
# Evict sequences that hit EOS or their length limit,
# freeing their slots for newcomers.
running = evict_finished(running)
running += admit_from_queue(slot_budget - len(running))
# One forward pass decodes one token for EVERY running
# sequence; finished ones are dropped on the next pass.
next_tokens = model.forward(running)
append_tokens(running, next_tokens)The GPU never runs a partially idle batch, because the batch is rebuilt every iteration from whatever is actually alive. New requests stop waiting for a “batch boundary” that no longer exists — they join at the next forward pass, which is milliseconds away. TTFT drops accordingly.
Orca paired this with selective batching: operations that are cheap to run across concatenated sequences (the big matrix multiplies in attention and feed-forward layers) are batched together for efficiency, while operations that are per-sequence by nature (sampling, normalization) are executed request by request. The paper reports throughput gains of up to 36.9x over FasterTransformer at equivalent per-token latency. A later public benchmark from Anyscale measuring static batching against vLLM’s continuous batching on a simulated live workload found a 23x throughput difference — your numbers will vary, but an order of magnitude under bursty, variable-length traffic is the norm, not the exception.
The second half: paged KV memory
Continuous batching answers “when does a request join the batch,” but a second problem blocks high concurrency: where does its KV cache live? Every sequence accumulates key/value tensors for every layer and every generated token. Preallocating KV space for a worst-case context length is how earlier systems did it, and it fragments GPU memory brutally — a request that will actually use 300 tokens reserves room for 32,000.
PagedAttention, vLLM’s core contribution, applies the same trick operating systems use for RAM: divide the KV cache into fixed-size blocks, allocate them on demand, and keep a per-sequence block table that maps logical token positions to physical blocks. Because allocation is now fine-grained, the scheduler can admit many more concurrent sequences into the same memory — which feeds directly back into continuous batching, since more admissible sequences means fuller batches at every iteration. The two techniques are a pair: iteration-level scheduling creates the demand for flexible memory, and paged memory makes high concurrency physically possible.
The knobs that actually matter
Every runtime exposes this machinery through a handful of settings. In vLLM, three matter most in practice:
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
--max-num-seqs 256 \
--max-num-batched-tokens 8192 \
--enable-chunked-prefillmax-num-seqs caps how many sequences may decode concurrently — the width of your batch. Raising it increases throughput until either KV memory or per-iteration latency becomes the binding constraint.
max-num-batched-tokens bounds the total token budget per forward pass. This is where chunked prefill enters: without it, a 30,000-token prompt must be processed as one enormous prefill step, and every decoding sequence in the batch stalls behind it — you see it as a spike in inter-token latency (ITL). Chunked prefill splits that prompt across several iterations, mixing its chunks with ongoing decodes so the decode stream never fully stops. If your users report “the stream hiccups whenever someone pastes a long document,” this is the setting you are missing.
One more behavior to understand before production: preemption. When KV memory runs out, the scheduler does not refuse new requests — it evicts a running sequence, discarding (or swapping out) its KV blocks to make room, then recomputes the sequence’s tokens from its prompt when capacity returns. Correct, but expensive: preempted work is redone, and the victim’s users watch their stream freeze. Sustained preemption activity means the server is over-admitted relative to its memory, and the fix is lowering max-num-seqs or giving the instance more headroom — not ignoring the counter.
What to monitor once it is on
Continuous batching changes which metrics predict user experience. Requests per second and average latency tell you almost nothing; the state of the queue and the shape of token latency tell you everything. At minimum, track the runtime’s running and waiting request gauges, its KV cache utilization percentage, queue-time distributions, and — separately for prefill and decode — the token latency percentiles (TTFT and ITL). A healthy server under load shows waiting requests above zero (work is queued, not lost), KV utilization high but not pinned, and an ITL p99 that stays flat as concurrency grows. Flat ITL under load is the whole game: it is the direct signature that iteration-level scheduling is absorbing arrivals without disturbing in-flight streams.
Where continuous batching does not help
It is worth knowing the boundaries. At very high request rates, the server saturates and becomes compute-bound — the batch is full at every iteration, and there is nothing left to schedule around; further gains must come from quantization, better kernels, or more hardware. Offline batch jobs with known, homogeneous sequence lengths are also fine with static batching: if every output is roughly the same length, there is no straggler problem to solve, and simpler code may win. The technique earns its keep precisely where production traffic lives — bursty arrivals, wildly variable output lengths, and interactive users who notice every stall.
If you run a serving stack released in the last few years, continuous batching is already on — the real work is tuning the concurrency limits, enabling chunked prefill for your workload mix, and watching the preemption counter. Start with the defaults, load-test with realistic length distributions, and let the ITL percentiles tell you when you have found the ceiling.