GPU – WorthPosting

AI News

vLLM Prefix Caching: The Prefill Optimization You’re Already Running

September 3, 2026 teliaz

Every request that hits an LLM server pays the same tax before the first generated token appears: the prompt must

Continue readingvLLM Prefix Caching: The Prefill Optimization You’re Already Running

Software Engineering

Speculative Decoding in vLLM: Draft, Verify, and Cut Latency Without Losing the Distribution

August 27, 2026August 27, 2026 teliaz

Large language model inference has an awkward performance profile: the GPU does enormous math, then waits. Every token requires a

Continue readingSpeculative Decoding in vLLM: Draft, Verify, and Cut Latency Without Losing the Distribution

Software Engineering

vLLM v0.23.0: Model Runner V2, Multi-Tier KV Offloading, and the Growing Rust Frontend

June 22, 2026June 23, 2026 teliaz

The vLLM v0.23.0 release landed last week with 408 commits from 200 contributors, and it packs several changes that directly

Continue readingvLLM v0.23.0: Model Runner V2, Multi-Tier KV Offloading, and the Growing Rust Frontend