Speculative Decoding Explained: How EAGLE-3 Makes LLMs 2-3x Faster Without Changing Outputs
Autoregressive decoding is the reason large language models feel slow: every token is generated by a full forward pass, and
Tag: vLLM
PagedAttention and the KV Cache: How OS Paging Explains Modern LLM Serving
When an LLM serves a 2,000-token response to a prompt of 10,000 tokens, it does something that looks absurd from
Continue readingPagedAttention and the KV Cache: How OS Paging Explains Modern LLM Serving
LLM Quantization Explained: GPTQ, AWQ, GGUF, and When Each One Wins
Serving a 7B model in fp16 takes roughly 14 GB of VRAM. A 70B model takes around 140 GB —
Continue readingLLM Quantization Explained: GPTQ, AWQ, GGUF, and When Each One Wins
Speculative Decoding: Getting 2-3x More Out of Every GPU Pass
Watch a GPU while a large language model generates text and you’ll see something strange: for most of every forward
Continue readingSpeculative Decoding: Getting 2-3x More Out of Every GPU Pass
vLLM Prefix Caching: The Prefill Optimization You’re Already Running
Every request that hits an LLM server pays the same tax before the first generated token appears: the prompt must
Continue readingvLLM Prefix Caching: The Prefill Optimization You’re Already Running
Speculative Decoding in vLLM: Draft, Verify, and Cut Latency Without Losing the Distribution
Large language model inference has an awkward performance profile: the GPU does enormous math, then waits. Every token requires a