When an LLM serves a 2,000-token response to a prompt of 10,000 tokens, it does something that looks absurd from
Continue readingPagedAttention and the KV Cache: How OS Paging Explains Modern LLM Serving
Every RAG system has two halves: a retriever that decides which text the model sees, and a generator that answers
Continue readingRAG Chunking Strategies in 2026: What the Benchmarks Actually Show
If you run a large language model behind a product, you have probably internalized an uncomfortable trade-off. The big frontier
Continue readingLLM Routing and Cascading: Send Every Query to the Cheapest Model That Can Handle It
Serving a 7B model in fp16 takes roughly 14 GB of VRAM. A 70B model takes around 140 GB —
Continue readingLLM Quantization Explained: GPTQ, AWQ, GGUF, and When Each One Wins
Watch a GPU while a large language model generates text and you’ll see something strange: for most of every forward
Continue readingSpeculative Decoding: Getting 2-3x More Out of Every GPU Pass
Every request that hits an LLM server pays the same tax before the first generated token appears: the prompt must
Continue readingvLLM Prefix Caching: The Prefill Optimization You’re Already Running
Ask an LLM for JSON and you will usually get JSON. “Usually” is the word that ruins your week. One
Continue readingConstrained Decoding: How Structured LLM Output Actually Works
Large language model inference has an awkward performance profile: the GPU does enormous math, then waits. Every token requires a
The commits that caught my attention this week range from a vector database that shrinks a 31 GB corpus into
The default answer to “make the agent better” is a better model. But a growing pile of evidence says that