The KV Cache: Why LLM Serving Is a Memory Problem

Every LLM serving stack lives or dies by one data structure most developers never see: the KV cache. It is the reason the second token of a response is fast, the reason a 70B model can hold more than two concurrent conversations on an 80GB GPU, and the reason your inference bill spikes when context windows grow. If you have ever wondered why serving a long-context request costs far more than the token count suggests, the KV cache is the answer.

This post walks through the KV cache from first principles: why autoregressive decoding needs it, how big it actually gets (with arithmetic you can run yourself), what PagedAttention changed, how prefix caching turns shared prompts into near-free input, and how architectural choices like grouped-query attention and multi-head latent attention shrink the cache before a single serving optimization is applied.

Why Autoregressive Decoding Needs a Cache

A transformer generates text one token at a time. To produce token n+1, attention at every layer must score the current query against the keys of all previous tokens and aggregate their values. The naive implementation recomputes keys and values for the entire sequence on every step, which turns generation into an O(n²) compute problem over the sequence length.

The fix is to cache the key and value projections of every previous token, per layer, per attention head. After each decoding step, the newly computed key and value vectors are appended to the cache, and the next step attends over the cache instead of recomputing history. That cache is the KV cache. It converts redundant recomputation into memory reads, which is exactly the trade modern GPUs are built for — but only up to the point where the cache itself becomes the bottleneck.

This is also why inference is described as memory-bound during decoding. Producing each token requires reading the model weights and the full KV cache from VRAM; the arithmetic per byte moved is small. The cache is not a side effect of serving — it is the dominant traffic on the memory bus during generation.

The Arithmetic: How Big Does the KV Cache Get?

The formula is simple enough to estimate on a napkin, and you should, because it tells you your ceiling on concurrency before anything else in the stack matters:

bytes = 2 (K and V)
      x layers
      x kv_heads
      x head_dim
      x tokens
      x bytes_per_element

For a Llama-2-7B-shaped model (32 layers, 32 heads, 128 head dimension) holding 4,096 tokens in fp16, the math is:

2 x 32 x 32 x 128 x 4096 x 2 bytes = 2,147,483,648 bytes ≈ 2 GiB

Two gibibytes for a single 4K-token sequence on a 7B model. The 80GB GPU that holds the 7B weights with room to spare can fit maybe 30 such sequences if nothing else consumed VRAM — and with contiguous allocation (more on that in a moment) the real number was historically far worse. Scale the context to 128K tokens and the cache for that one sequence is over 60GB: the weights stop being the largest thing on the device.

Two levers dominate this formula: the token count (linear, and brutal at long context) and the number of key/value heads. That second lever is where architecture comes in.

GQA and MQA: Fewer Heads, Same Attention

Multi-query attention (MQA) lets all query heads share a single key/value head, cutting the cache by a factor equal to the number of heads. Grouped-query attention (GQA) interpolates: query heads are divided into groups, and each group shares one KV head. Llama-2-70B uses GQA with 8 KV heads against 64 query heads — an 8x cache reduction with a quality cost that proved negligible in practice.

GQA is the reason a 70B model with a 128K context is servable at all. It is also a reminder that cache efficiency is decided at training time. You cannot retrofit GQA onto a multi-head checkpoint; if you are serving a MHA model, the serving stack has to do the work instead.

MLA: Compressing the Cache Itself

Multi-head latent attention (MLA), introduced in DeepSeek-V2, takes a different approach: instead of sharing heads, it compresses keys and values into a low-rank latent vector and stores only that. The cache per token drops to the latent dimension (512 in DeepSeek-V3) plus a small per-head rope component, rather than the full heads x head_dim expansion. At decode time, up-projection matrices reconstruct full keys and values from the latent. The reported reduction is on the order of 90%+ versus the equivalent multi-head cache, which is why DeepSeek models are unusually cheap to serve relative to their size.

The trade-off is compute: the reconstruction adds FLOPs per step, a cost that only pays off because decoding is memory-bound. And MLA interacts poorly with naive prefix reuse, since the cached latent is not directly compatible with RoPE-keyed attention — serving stacks needed specific support before MLA models could benefit from prefix caching.

PagedAttention: Fixing the Allocation Problem

Before paged attention, serving systems allocated one contiguous VRAM region per request, sized for the maximum possible sequence length. Requests arrive with wildly different actual lengths, so the gap between reserved and used memory was wasted: the original vLLM analysis measured 60-80% waste in real traces, and systems that preallocated for worst-case context windows effectively served a handful of concurrent requests per GPU.

PagedAttention borrows virtual memory from operating systems. The cache is chopped into fixed-size blocks (commonly 16 tokens per block), a per-request block table maps logical token positions to physical blocks, and attention kernels gather K/V through the table. Allocation becomes on-demand: a request that ends up generating 300 tokens never touches the memory reserved for 32,768.

The payoff is measured in concurrency, not latency: vLLM reports under 4% memory waste versus the 60-80% baseline, which translates directly into more sequences per batch — historically 2-4x throughput at comparable latency. The costs are engineering complexity in the block manager and a small overhead for indirect addressing in the attention kernel.

Prefix Caching: Reuse Across Requests

Paging makes the cache compact; prefix caching makes it reusable. Long system prompts, few-shot examples, retrieved documents, and multi-turn conversation history are shared across requests — and the KV blocks computed for a shared prefix are bit-identical every time. Prefix caching keeps those blocks alive after a request finishes and lets the next request with the same prefix reference them instead of recomputing.

Two implementation details matter if you are tuning this in production:

Block granularity. Hits are counted per full block. Two prompts that share the first 4,000 tokens of a 4,096-token system prompt but diverge in the last block get credit only for the complete shared blocks. Padding or normalizing your prompt so the shared part ends on a block boundary measurably improves hit rates.
Multi-turn conversations are the ideal workload: every follow-up turn’s prompt is a strict extension of the previous one, so nearly the entire prompt hits cache and time-to-first-token collapses.

One security note: because a cache hit is faster than a miss, timing differences can leak whether some other request already processed a given prefix. Systems address this by salting the cache key per tenant, so one customer’s cached context can never serve another’s.

What This Means for Your Workloads

RAG: the retrieved documents usually dominate the prompt, and the same documents recur across queries. Cache-friendly retrieval — keeping chunk boundaries stable so cached blocks stay valid — often beats further reranking effort.
Agents: the append-only conversation history is the perfect prefix pattern. Append, never rewrite: if your orchestration layer reshuffles or re-serializes history between turns, you are invalidating your own cache.
Batch evaluation: a shared few-shot preamble across thousands of requests means the preamble is computed once. Structure prompts so the shared part comes first, byte-identical every time.

The general principle: the KV cache rewards workflows where requests share long, stable prefixes and punishes anything that perturbs the head of the prompt. A timestamp at the start of your system prompt can silently disable caching for every request of the day.

Wrapping Up

The KV cache is where LLM serving economics are decided. Architecture (GQA, MLA) sets the per-token cost; PagedAttention eliminates allocation waste; prefix caching amortizes shared context across requests; quantization and offloading stretch the budget further. None of these require touching model weights — but the highest-leverage move is usually on the application side: design prompts and conversation flows so that context is shared and append-only, and the serving stack does the rest.

If you want to go deeper, the vLLM documentation on automatic prefix caching explains the hash-based block reuse design, the model documentation for Llama 2 shows a GQA configuration in concrete terms, and the MLA write-up in the DeepSeek-V2 report covers the latent compression math in full.