The KV Cache: Why Long Contexts Eat Your GPU Memory

If you have ever wondered why serving a 70B model needs a GPU with 80 GB of memory when the weights themselves are only about 40 GB, the answer is not the model. It is the bookkeeping. Every token an LLM generates requires attention scores against every token before it, and recomputing those scores from scratch at every step would make inference quadratically slower than it already is. So inference engines keep the intermediate state — the keys and values each layer produced — in GPU memory for the lifetime of the request. That state is the KV cache, and it is the single largest consumer of memory in a production inference stack.

Understanding the KV cache changes how you read every serving benchmark. It explains why long contexts are billed at a premium, why your GPU can only fit a handful of concurrent requests, and why architectures like grouped-query attention exist at all. This post walks through what the cache actually stores, how to compute its size for a real model, and the three levers — attention architecture, paging, and reuse — that serving systems pull to keep it under control.

Why the cache exists at all

A transformer generates text one token at a time. At each step, the newest token is turned into a query vector, and that query is compared against the keys and values of every previous token in every attention layer. The crucial observation is that the keys and values for tokens 1 through N do not change when token N+1 arrives — earlier tokens have already been fully processed. Only the new token produces new key and value vectors.

That is what the cache stores: for every layer, for every key/value head, one key vector and one value vector per token processed so far. Without the cache, generating token N+1 would mean re-running all N previous tokens through the model’s attention computations. With the cache, each step reads the stored vectors and appends one new entry per layer. The trade is compute for memory: a constant amount of GPU memory per token, held for as long as the request lives.

The size formula

The cache size per token follows directly from what is stored. Multiply two vectors (one key, one value) by the number of layers, by the number of key/value heads, by the head dimension, and by the bytes per element:

bytes_per_token = 2 * num_layers * num_kv_heads * head_dim * dtype_bytes

total_cache = bytes_per_token * sequence_length * batch_size

Run this against a real model and the numbers get uncomfortable fast. Take a Llama 3.1 70B-style architecture: 80 layers, 64 query heads, 8 key/value heads, a head dimension of 128, and 16-bit weights. Each token costs 2 × 80 × 8 × 128 × 2 bytes, which is 320 KiB per token. A single 8,192-token request therefore holds 2.5 GiB of cache, and a full 128k-context request holds 40 GiB — nearly half of an 80 GB accelerator, before you have served a second user.

You can verify the arithmetic yourself with a few lines of Python:

GiB = 1024 ** 3

def kv_cache_gib(layers, kv_heads, head_dim, seq_len, dtype_bytes=2):
    bytes_per_token = 2 * layers * kv_heads * head_dim * dtype_bytes
    return bytes_per_token * seq_len / GiB

# Llama 3.1 70B-style geometry
print(kv_cache_gib(80, 8, 128, 8_192))     # 2.5 GiB per request
print(kv_cache_gib(80, 8, 128, 131_072))   # 40.0 GiB at full context

# The same model without grouped-query attention (64 KV heads):
print(kv_cache_gib(80, 64, 128, 131_072))  # 320.0 GiB -- does not fit

Two consequences fall out of the formula immediately. First, cache memory scales linearly with sequence length, so doubling the context window doubles the memory bill per request. Second, memory left over after the weights determines your concurrency ceiling: with roughly 67 GiB free after a 70B model’s weights, you can host about 26 concurrent 8k-token requests — or three requests at full context.

Shrinking the cache in the architecture: MQA, GQA, MLA

Each term in the formula has been attacked by an attention variant. Multi-head attention gives every query head its own key and value head, which is the expensive baseline. Multi-query attention shares a single key/value head across all query heads, dividing the cache by the head count — an aggressive cut that can cost quality. Grouped-query attention is the middle ground: groups of query heads share one key/value head. Most current open-weight models use it. Llama 3.1 70B has 64 query heads sharing 8 key/value heads, which is exactly the 8× gap between the 320 GiB and 40 GiB figures above.

The more recent entry compresses the vectors themselves. Multi-head latent attention, introduced with DeepSeek-V2, caches one compressed latent vector per token per layer (plus a small positional key) and reconstructs keys and values from it at attention time. DeepSeek-V3 stores 576 elements per token per layer this way, against 2 × 8 × 128 = 2,048 for a grouped-query model of comparable scale — roughly a 3.5× reduction that compounds over long contexts. The cost is a decode-time reconstruction step and a more intricate training setup.

Managing the cache at serving time: paging

Fixing the architecture fixes bytes per token, but the serving engine still has to decide how well those bytes get used. Early serving systems pre-allocated a contiguous cache region sized for the model’s maximum context length for every request. Since almost no request actually reaches that maximum, most of the reserved memory sat wasted — the PagedAttention paper measured utilization in existing systems as low as 20–40 percent. Requests were also queued rather than batched dynamically, because their reserved regions could not be reshuffled.

The fix borrows a page from operating systems, literally. Instead of contiguous reservations, the cache lives in fixed-size blocks, and a per-request block table maps logical positions to physical blocks, exactly like virtual memory pages. Fragments shrink to at most one block per request, and a shared prefix (a system prompt, for example) can be referenced by multiple requests through copy-on-write semantics. The vLLM design notes describe the mechanics; the reported result was 2–4× higher throughput than contemporaneous systems at the same latency, with near-zero cache waste.

The same machinery powers prefix caching more broadly. If two requests share a 2,000-token system prompt, the second request reuses the first prompt’s blocks instead of recomputing them — turning prefill work into a memory lookup. Production operators extend this into a hierarchy of GPU memory, host RAM, and cluster-wide caches, so a prefix computed on one node can be reused on another.

Practical implications

A few rules of thumb follow from the math, and they are useful when you are sizing deployments or reading a serving dashboard:

  • Compute your own numbers. The formula needs only the config: layers, key/value heads, head dimension, and dtype. Do not trust “supports 128k context” marketing without checking whether the memory exists to serve even one request at that length.
  • Concurrency and context length are the same budget. Every token in flight, across every request, buys the same bytes. A deployment that fits 26 concurrent 8k requests fits about 3 at 128k. Serving stacks enforce this with admission control, queuing or preempting requests when the pool of blocks runs dry.
  • Cache quantization is a real lever with a real cost. Storing keys and values in 8-bit instead of 16-bit halves the cache, roughly doubling the concurrency ceiling, at the price of some attention precision. It is a different trade from weight quantization, and the two compose.
  • Prefill and decode stress different resources. Prefill is compute-bound and scales with prompt length; decode is memory-bandwidth-bound and scales with cache size. This asymmetry is why production platforms increasingly run the two phases on separate pools of hardware.

The KV cache is not an implementation detail you can ignore — it is the resource that determines how many users a GPU can serve and how long their contexts can be. The next time a model card advertises a giant context window, run the formula before you run the model. If the cache does not fit, the context window is a spec sheet number, not a capability.

Leave a Reply

Your email address will not be published. Required fields are marked *