Serving a large language model to thousands of concurrent users is, underneath all the marketing, a memory management problem. Every active request holds a KV cache: the stored attention keys and values for every token in its context. The bigger your context window and the higher your concurrency, the more HBM bytes this cache consumes — and HBM is the scarcest resource in the entire serving stack. Most model release announcements compete on parameters and benchmark scores. DeepSeek’s latest report quietly competes on something more concrete: how few bytes per token the cache occupies. The model is DeepSeek-V4.1-Flash, and its technical report, “Pushing the Limits of KV Cache Compression,” reads less like a model card and more like a storage systems paper. That is exactly why it is worth an engineer’s attention.
This post walks through the architecture from the perspective of someone who has debugged serving clusters, not trained models. The numbers matter here: the report claims the always-resident global KV cache has been compressed to roughly 890 bytes per token — about a quarter of the previous V4-Flash generation and hundreds of times smaller than the original DeepSeek-V1’s cache footprint. Multiply that by a million-token context and a few hundred concurrent requests, and you can see why byte-level compression is the difference between a feasible deployment and a non-starter.
The KV cache is the real budget
A quick refresher on why the cache dominates serving economics. During autoregressive decoding, the model must attend to every previous token. Recomputing keys and values for the whole prefix at every step would be quadratic, so every inference engine caches them. The cache size scales with three factors: number of layers, number of KV heads times head dimension, and sequence length. The first two are fixed per model; sequence length is whatever the user throws at you. This is why a “1M token context” claim means nothing until you ask how many bytes per token the cache costs.
Before this generation, the report identifies three orthogonal dimensions for shrinking that cache, and V4.1-Flash is notable for combining all three rather than picking one:
- Entry size — reduce bytes per cache entry. Grouped-query attention (GQA) reduces the number of KV heads; multi-head latent attention (MLA) projects all heads into a shared low-rank latent. Both shrink the per-entry footprint.
- Sequence dimension — compress many tokens into one entry. This is the territory of sparse attention with cache selection: keep representations for only the tokens that actually matter, not all of them.
- Layer dimension — let some layers reuse the cache and selection results of other layers, instead of every layer maintaining its own full copy.
The interesting engineering insight is the multiplication: a 4x entry reduction combined with 4x sequence-level sparsity and layer-level reuse compounds into savings no single technique could deliver. The report’s headline numbers — 1/4 of V4-Flash’s global cache per token in HBM, roughly 1/8 for the persistent tier used for prefix reuse — come from stacking these dimensions.
HySparse: layers that share instead of duplicating
The component doing the heavy lifting on the sequence and layer dimensions is the sparse attention mechanism, referred to in the report as HySparse. The design keeps full-attention layers as anchors, then lets sparse layers reuse the KV cache maintained by those dense layers instead of holding their own copy. The crucial part is how the reuse works — there are two operating modes, and the distinction matters if you are reasoning about what the cache actually contains:
- Full mode — a layer computes its own main KV entries and its own indexer queries, projects the indexer keys from the main KV, and runs the complete token-selection process to produce a fresh set of top-k indexes.
- Reindex mode — a layer reuses the main KV and indexer keys of a previous layer, but keeps its own indexer queries. It re-scores tokens with those queries and selects its own top-k. The cache is shared; the selection is independent.
The reindex trick is the kind of compromise that only looks obvious in retrospect. Early cache-selection schemes forced all layers to attend to the same tokens, which hurts quality whenever a layer needs something its neighbors did not select. Reindex mode keeps the storage benefit of sharing while decoupling what each layer reads from the shared pool. The trade is extra compute for scoring — bytes are cheap to share, but each sparse layer still pays attention-dot-products over the candidates. For a serving deployment, that is usually the right way around: HBM capacity is the hard constraint, FLOPs have headroom.
The model behind the cache numbers
Architecture-wise, V4.1-Flash is a multimodal mixture-of-experts model: a 552B-parameter backbone plus 196B “Engram” parameters, pretrained on a 45T-token multimodal corpus with native support for 1M-token contexts. Activation is asymmetric across phases — around 8B parameters per token during prefill and about 16B during decode. The MoE asymmetry is worth noting: decode activates more experts than prefill, reflecting the fact that generation quality is more sensitive to expert coverage than ingest.
Cache numbers put this in context. The report’s per-generation comparison shows the always-resident global KV shrinking from roughly 389KB per token in DeepSeek-V1 to under 1KB per token in V4.1-Flash. On top of the in-HBM cache, there is a tiered story: the persistent cache (SSD and host memory, used for long-lived prefix reuse across sessions) gets its own compression treatment, which we will look at next.
Tiered storage for attention state
Here is where the paper stops reading like an ML report and starts reading like a database storage-engine design. The persistent cache — the portion kept around for prefix reuse so returning sessions do not re-prefill everything — previously had to hold both global attention state and sliding-window (SWA) attention state. The report points out a lifecycle mismatch: global KV has long-tail reuse value and justifies durable storage, but sliding-window KV is only useful within an active session and becomes dead data the moment the session ends.
The fix is a classic tiering decision. V4.1-Flash moves sliding-window KV out of the persistent tier entirely and into a distributed memory pool built from about 10% of host DRAM on each machine, with a TTL of only a few minutes. Global KV stays on SSD with a lifecycle measured in tens of hours, where the report cites at least 72 hours of retention. Different data, different access patterns, different storage classes — the same reasoning you would apply to a write-through cache versus durable row storage.
Another piece worth stealing for your own systems is what the report calls bounded replay. Instead of assuming a cached prefix is valid forever, the system caps how much recomputation it is willing to do when a cached entry turns out to be stale or evicted. It is the inference-serving analog of a read-repair bound: cache misses degrade gracefully instead of turning into full re-prefill storms that would crush latency SLOs. Anyone who has run a CDN or a database replica set will recognize the pattern immediately.
FP4 for the cache
Quantization usually gets discussed for weights. V4.1-Flash applies FP4 to the KV cache itself — compressing the stored keys and values, not just the model parameters. This is the entry-size dimension pushed to its floor. The risk with ultra-low precision caches is accumulated error across long sequences: attention scores computed against lossy keys drift, and quality decays as context grows. The report’s claim is that the combination of latent-projection (MLA-style) caching plus careful per-component precision allocation keeps quality within useful bounds while hitting the byte targets. For serving engineers, the takeaway is that cache precision is now a tunable dial on the same level as batch size and tensor parallelism — and leaving it at FP16 is leaving capacity on the table.
What to actually take from this
Three points are worth carrying back to ordinary engineering work, whether or not you ever deploy a 552B-parameter model:
- Compress along multiple axes, not one. The cache gains came from multiplying entry-size, sequence-level, and layer-level reductions. Any hot resource in your system — connection pools, row caches, index pages — usually has more than one compressible dimension if you look for them.
- Match storage tier to data lifecycle. Sliding-window state belongs in short-TTL memory; global state belongs on durable media. Misplaced lifecycles are how caches end up full of dead data, in inference clusters just like in application backends.
- Bound your repair costs. Bounded replay turns unbounded cache-miss penalties into a budgeted expense. It is the same reason retry budgets and circuit breakers exist elsewhere in distributed systems.
The broader signal is that the frontier of LLM efficiency has moved. A few generations ago the race was about training compute and parameter counts; now the decisive constraint is memory bandwidth and bytes-per-token residency, and the teams winning that race are the ones treating attention state as a first-class storage problem. The full technical report is on arXiv, and DeepSeek maintains its code and model releases on GitHub. If you operate inference infrastructure, read it as a storage-engine paper — because that is what it is.