Every token an LLM emits costs a full forward pass. The GPU reads hundreds of gigabytes of weights, computes one
Continue readingSpeculative Decoding: The Math, the Drafters, and When the Bet Pays Off
Every LLM serving stack lives or dies by one data structure most developers never see: the KV cache. It is
Continue readingThe KV Cache: Why LLM Serving Is a Memory Problem
Every LLM application has two knobs that get blamed for everything: temperature and top-p. When outputs are boring, someone raises
Continue readingTemperature, Top-p, Top-k, and Min-p: How the LLM Sampler Actually Works
Four-bit weights used to be a hobbyist compromise — the thing you did to squeeze a 70B model onto a
Continue reading4-Bit by Default: Choosing Between GPTQ, AWQ, GGUF, and MXFP4 in 2026
Serving a large language model to thousands of concurrent users is, underneath all the marketing, a memory management problem. Every
Continue readingDeepSeek-V4.1-Flash: Why the Most Interesting AI Paper This Month Is About Storage
Autoregressive decoding is the reason large language models feel slow: every token is generated by a full forward pass, and
When an LLM serves a 2,000-token response to a prompt of 10,000 tokens, it does something that looks absurd from
Continue readingPagedAttention and the KV Cache: How OS Paging Explains Modern LLM Serving
Serving a 7B model in fp16 takes roughly 14 GB of VRAM. A 70B model takes around 140 GB —
Continue readingLLM Quantization Explained: GPTQ, AWQ, GGUF, and When Each One Wins
Watch a GPU while a large language model generates text and you’ll see something strange: for most of every forward
Continue readingSpeculative Decoding: Getting 2-3x More Out of Every GPU Pass
Every request that hits an LLM server pays the same tax before the first generated token appears: the prompt must
Continue readingvLLM Prefix Caching: The Prefill Optimization You’re Already Running