Every conversation with a chatbot hides a growing accounting problem. Each token the model generates must attend to every token before it — so every prior token must keep its key and value vectors in memory. Across a batch of concurrent users and a long context window, the cache that makes generation fast becomes the dominant consumer of GPU memory, often exceeding the model weights themselves. Understanding this cache explains most of the configuration decisions you’ll make when deploying an LLM. We’ll build it from first principles, price it with verifiable numbers, and walk through the techniques — paging, continuous batching, and prefix reuse — that turn those numbers from a hard ceiling into a tunable budget.
Why Attention Forces the Issue
Autoregressive generation produces one token per forward pass. To generate token n+1, attention needs the key and value projections of every token up to and including n. The naive implementation recomputes those projections for the whole sequence on every step — correct, but quadratic in compute, and almost entirely wasted work, because those projections don’t change. They can’t, because of the causal mask: each token’s key and value depend only on the tokens at or before its own position, so a new token appended at the end never alters them. The model can compute each token’s K and V vectors exactly once and store them for the rest of the generation. That store is the KV cache, and it’s a pure trade: per-step compute drops from quadratic to linear, while memory grows linearly with sequence length. You are permanently exchanging VRAM for speed.
Pricing the Cache
The cache is big, and the formula for exactly how big is short. Every layer stores both K and V for each token, so:
bytes per token = 2 (K and V) × layers × KV heads × head dimension × bytes per element
Run the numbers for a 7B-class model with multi-head attention — 32 layers, 32 KV heads, head dimension 128, FP16 weights:
def kv_bytes_per_token(num_layers, num_kv_heads, head_dim, dtype_bytes):
# The leading 2 accounts for the separate K and V tensors.
return 2 * num_layers * num_kv_heads * head_dim * dtype_bytes
mha_7b = kv_bytes_per_token(num_layers=32, num_kv_heads=32,
head_dim=128, dtype_bytes=2)
print(mha_7b / 1024) # 512.0 KB per token, all layers combined
print(mha_7b * 4096 / 2**30) # 2.0 GB for one 4096-token sequence
gqa_70b = kv_bytes_per_token(num_layers=80, num_kv_heads=8,
head_dim=128, dtype_bytes=2)
print(gqa_70b / 1024) # 320.0 KB per tokenHalf a megabyte per token sounds trivial until you multiply. A single 4,096-token conversation costs roughly 2 GB of cache — for a model whose weights only occupy about 14 GB in FP16. Batch 32 such conversations on one GPU and the cache alone demands on the order of 64 GB. Long-context serving is a memory problem before it is a compute problem: the weights are rented once; the cache is rented per sequence, per token, for as long as the sequence lives.
It also explains why grouped-query attention became near-universal in recent open models. GQA shares each key/value head across a group of query heads, shrinking the KV-head count without a proportional quality loss. A 70B model using 8 KV heads instead of 64 cuts the per-token cache by 8× — from about 2.5 MB down to 320 KB per token — often the difference between serving a 4K context and a 128K one on the same hardware. Multi-head latent attention compresses the cache further still. When you evaluate model checkpoints for serving, treat the KV-head count and head dimension as capacity specifications, not trivia.
Prefill, Decode, and Two Different Bottlenecks
Inference has two phases, and the cache behaves differently in each. Prefill processes the entire prompt in one pass, computing and storing its K/V vectors; with all tokens known upfront, the work is a handful of large matrix multiplications — compute-bound and efficient. Time-to-first-token is mostly prefill time. Decode then generates one token per step, each new query reading against the whole cached history: a small amount of math that repeatedly streams a large amount of memory. That’s memory-bandwidth-bound — tokens arrive slower than the first one, and per-sequence decode throughput is dominated by how fast the hardware moves cache bytes, not by FLOPs. Everything that follows is about managing the memory side of that trade, because naive cache allocation wastes most of it.
Fragmentation: Paying for Context You Never Use
The obvious implementation reserves, at request arrival, one contiguous VRAM region sized for the model’s maximum sequence length. Simple to reason about, catastrophic in practice: a request that eventually uses 200 tokens occupies space for 8,192 for its entire lifetime, and because the region is contiguous, nothing else can use the gap. Waste splits into over-reservation (allocated-but-unused slots) and external fragmentation (free memory too scattered to serve a new request). Measured overhead under this scheme routinely lands in the 60–80% range, capping concurrency at a fraction of what the hardware could hold — and concurrency is where throughput lives.
PagedAttention: Virtual Memory for the GPU
The PagedAttention paper borrowed the fix directly from operating systems. Instead of one contiguous reservation per sequence, the cache pool is chopped into fixed-size physical blocks — 16 tokens each, by default in vLLM’s design docs — and each sequence keeps a logical block table mapping its tokens to blocks scattered anywhere in GPU memory. Blocks are claimed on demand as tokens arrive; waste per sequence shrinks to the partially-filled last block, and measured pool overhead drops to under 4%.
The block table buys a second win: cheap sharing. When a request asks for parallel sampling or beam search, the forked candidates share the parent’s physical blocks by reference and only allocate new blocks for tokens that diverge — copy-on-write, as a forked process shares pages. Prompts that are identical across requests can share their prefix blocks the same way, which is the foundation of prefix caching.
The cost of indirection is real but modest: the attention kernel must gather K/V from non-contiguous blocks via the block table on each step. At high concurrency the memory savings dominate and throughput wins outright; at batch size one, contiguous layouts retain a small latency edge.
Keeping the GPU Fed: Continuous Batching and Chunked Prefill
Memory management solves only half the utilization problem; scheduling solves the other half. Under static batching, a group of requests is formed once and the whole batch runs until the longest sequence finishes — so thirty requests that complete at 50 tokens idle the GPU behind two stragglers running to 500. Iteration-level scheduling, usually called continuous batching, re-evaluates the batch after every forward pass: finished sequences exit and release their blocks, waiting requests join immediately. Under realistic mixed-length traffic, utilization under static batching can fall into the 20–40% range.
One wrinkle: a very long prompt’s prefill is a huge compute job, and running it as one unit stalls every decode in the batch behind it. Chunked prefill splits the prompt into pieces scheduled across iterations, letting decode tokens interleave between chunks — and since prefill chunks make the forward pass compute-bound, the memory-bound decode work effectively piggybacks on batch dimensions that are already large. Engines such as vLLM, TensorRT-LLM, and SGLang combine paging, continuous batching, and chunked prefill as the default serving posture.
Stretching the Budget Further
Two more levers sit on top of the fundamentals:
- Prefix caching. Chat applications resend the system prompt and prior turns on every request. A shared-prefix cache keeps the K/V blocks for the common prefix keyed by its hash, so each new request only prefills its true suffix. Because blocks are shared by reference, ten thousand sessions with the same instructions pay for one copy.
- Cache quantization. Storing K/V in FP8 instead of FP16 halves the bytes per token, doubling effective context or concurrency for a given VRAM budget. Attention reads are more noise-tolerant than weight matmuls, so quality degradation is usually small — but it is per-model and per-workload, so benchmark before you commit. Compute the cache at the quantized dtype in the formula above.
What This Means for Your Deployments
- Compute your cache budget before picking hardware. Per-token bytes × expected concurrent tokens tells you whether a GPU fits your workload.
- Prefer GQA/MLA checkpoints for serving. The lower KV-head count multiplies directly into more concurrent sessions per GPU.
- Cap context honestly. A 128K context you never use still costs scheduling headroom; a realistic max sequence length keeps more of the pool serving real traffic.
- Serve with a paged, continuous-batching engine. A naive generate loop leaves the majority of your VRAM and GPU time on the table.
- Watch cache pressure as the capacity signal. Engines report preemption and block-pool waiting when sequences get evicted or queued for memory — that should drive scaling decisions before latency does.
Wrapping Up
The KV cache is the quiet contract at the center of LLM serving: you trade memory for speed, and the bill grows with every token in every live conversation. Once you can price that bill — two tensors, per layer, per KV head, per token, per byte-width — the design of modern inference engines stops looking like a pile of tricks and starts looking like one coherent idea: manage cache memory the way operating systems manage RAM, schedule compute in increments small enough to never idle, and never store a byte twice.