LLM Quantization Explained: GPTQ, AWQ, GGUF, and When Each One Wins
Serving a 7B model in fp16 takes roughly 14 GB of VRAM. A 70B model takes around 140 GB —
Continue readingLLM Quantization Explained: GPTQ, AWQ, GGUF, and When Each One Wins
Tag: MLOps
Speculative Decoding: Getting 2-3x More Out of Every GPU Pass
Watch a GPU while a large language model generates text and you’ll see something strange: for most of every forward
Continue readingSpeculative Decoding: Getting 2-3x More Out of Every GPU Pass
vLLM Prefix Caching: The Prefill Optimization You’re Already Running
Every request that hits an LLM server pays the same tax before the first generated token appears: the prompt must
Continue readingvLLM Prefix Caching: The Prefill Optimization You’re Already Running
Constrained Decoding: How Structured LLM Output Actually Works
Ask an LLM for JSON and you will usually get JSON. “Usually” is the word that ruins your week. One
Continue readingConstrained Decoding: How Structured LLM Output Actually Works
Speculative Decoding in vLLM: Draft, Verify, and Cut Latency Without Losing the Distribution
Large language model inference has an awkward performance profile: the GPU does enormous math, then waits. Every token requires a
Building LLM Evaluation Pipelines: Metrics, Frameworks, and CI Integration
Shipping an LLM-powered feature without an evaluation pipeline is like deploying a microservice without health checks — it might work
Continue readingBuilding LLM Evaluation Pipelines: Metrics, Frameworks, and CI Integration
Training-Inference Mismatch: Why Your LLM Reinforcement Learning Is Optimizing the Wrong Policy
Reinforcement learning has become the defining ingredient of modern LLM post-training. GRPO, PPO, and their variants drive the reasoning capabilities
vLLM v0.23.0: Model Runner V2, Multi-Tier KV Offloading, and the Growing Rust Frontend
The vLLM v0.23.0 release landed last week with 408 commits from 200 contributors, and it packs several changes that directly
Beyond Naive RAG: 4 Advanced Patterns That Actually Work in Production
The first version of any RAG pipeline usually looks the same: embed a query, search a vector store, stuff the
Continue readingBeyond Naive RAG: 4 Advanced Patterns That Actually Work in Production