Every request that hits an LLM server pays the same tax before the first generated token appears: the prompt must be run through the model to fill the KV cache. For a chat application whose system prompt alone runs to ten thousand tokens, that tax is paid on every single message. vLLM’s automatic prefix caching (APC) eliminates it — and if you run a recent version of vLLM, there is a good chance it is already on. This post covers how it works, when it helps, and the cases where it quietly does nothing.
The Prefill Tax
LLM inference has two phases. Prefill processes the prompt and produces the key/value tensors the attention layers need; decode generates output tokens one at a time, reusing those tensors. Decode is bounded by memory bandwidth. Prefill is compute-bound, and its cost scales with prompt length — a request with an 8K-token prompt spends far more GPU-seconds processing the prompt than generating a 200-token answer.
The observation behind prefix caching is that this work is almost always redundant. Prompts share structure: a fixed system prompt, a long document pasted at the top of a RAG query, the growing conversation history of a chat session. Two requests that share a token prefix compute identical KV tensors for that shared portion. Caching those tensors and reusing them skips the prefill of the shared prefix entirely — and because the KV values for given tokens at given positions are deterministic, reuse does not change model outputs at all.
How vLLM Decides What to Reuse
The KV cache in vLLM is divided into blocks (16 tokens by default). vLLM hashes each full block using three components: the hash of the parent block, the exact token IDs in the block, and extra values that make the block unique — LoRA adapter IDs, hashes of any multimodal inputs, and an optional per-request cache salt. Chaining hashes through the parent means block N’s identity depends on the entire prefix before it, so two blocks with the same tokens at different positions never collide. Since v0.11, the default hashing algorithm is sha256, which closed the collision risks of the earlier scheme.
Only full blocks are cached. A request whose prompt is 40 tokens plus 5 leaves the trailing 5-token partial block uncached — prefix reuse happens at block granularity, so the last partial block of every prompt is always recomputed.
When a new request arrives, the scheduler hashes its prompt blocks and looks them up in the cache. Matched blocks are “touched” — their reference count goes up and they are pinned against eviction — and the request only pays prefill for the remaining unmatched tokens. When a request finishes, its blocks are freed into an LRU free queue, appended in reverse order so the tail blocks (least likely to be shared) are evicted first. Memory pressure simply evicts from this queue; there is no separate configuration for cache capacity versus KV cache capacity. The cache and the running requests share the same GPU memory pool, sized by your gpu_memory_utilization setting.
Enabling and Configuring It
In the v1 engine, prefix caching is on by default for every model that supports it. You only need to think about it when a model doesn’t support it, or when you want to turn it off:
# APC is enabled by default in vLLM v1 for supported models.
# Explicitly disable it (rare, but valid for some workloads):
vllm serve meta-llama/Llama-3.1-8B-Instruct --no-enable-prefix-caching
# Choose a reproducible hashing algorithm (default is sha256):
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--prefix-caching-hash-algo sha256_cborThe hash algorithm choices are sha256 (default, cryptographically secure but not byte-reproducible across Python versions), sha256_cbor (reproducible and cross-language compatible via CBOR serialization), and xxhash/xxhash_cbor (faster, non-cryptographic — the docs warn that this theoretically increases collision risk, which in a multi-tenant deployment is a privacy consideration, not just a correctness footnote).
For offline inference the engine argument is enable_prefix_caching=True. On the serving side, the OpenAI-compatible server also exposes a /reset_prefix_cache administrative endpoint that flushes cached blocks — useful after a model reload or when you know the cached prefixes are stale.
Multi-Tenancy and the cache_salt
Shared caches leak information through timing. If tenant A’s 50K-token prompt is cached, tenant B can send candidate prefixes and measure prefill latency to infer what A sent — shorter prefill means “match”. vLLM addresses this with request-level salting:
{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"}
],
"cache_salt": "tenant-a-salt"
}The salt is mixed into the hash of the first block, so requests only reuse cache within the same salt group. Requests without a salt share the default namespace. If you serve multiple customers from one deployment, this is the mechanism that lets you keep prefix reuse within each trust boundary.
When It Helps — and When It Does Nothing
APC is a prefill optimization. The official documentation is refreshingly honest about its limits, which map directly onto workload choice:
- Great fit: long system prompts shared across all requests; multi-turn chat, where each round’s prompt is the previous round’s prompt plus a few tokens — the entire history hits cache; repeated queries against the same long document (manuals, contracts, annual reports).
- Partial fit: RAG pipelines, if the retrieved context is stable across queries. If retrieved documents arrive in a different order each time, the shared prefix breaks at the first difference and the hit rate collapses.
- No benefit: generation-heavy workloads. APC never speeds up decode, so a service producing long answers from short, unshared prompts gets nothing. Likewise, prompts that share no prefix — a stream of independent one-shot requests with different instructions — cannot reuse anything.
Prefix matching is strict: the cache breaks at the first token that differs. That means prompt layout discipline matters. Put static content first and per-request content last. A system prompt with a timestamp embedded in the middle poisons every block after the timestamp; put the timestamp at the end and everything before it stays reusable.
To measure what you’re getting, watch the cached_tokens field returned per usage object by the OpenAI-compatible API — it reports how many prompt tokens were served from cache — and compare end-to-end time-to-first-token before and after enabling or restructuring prompts.
The Takeaway
Automatic prefix caching is one of the few inference optimizations that changes no outputs, requires no model changes, and ships enabled by default. The work left is architectural, not configuration: structure prompts so the expensive parts are shared, keep volatile content at the tail, use cache_salt where tenants must be isolated, and verify gains with cached_tokens rather than assuming. For chat and system-prompt-heavy workloads, it is the difference between paying full price for prefill on every request and paying it once.