Ask an inference server to answer one question and you get one GPU doing something odd: running a matrix multiplication
Continue readingContinuous Batching: The Scheduling Model Behind Every LLM Serving Stack
Every conversation with a chatbot hides a growing accounting problem. Each token the model generates must attend to every token
Continue readingThe KV Cache: The Memory Bill Behind Every Token Your Model Generates
Mixture-of-Experts models dominate the frontier right now, and almost every explanation stops at the same level: “a router picks a
Continue readingInside MoE Routing: How the Gate Decides Which Experts See Your Token
Ask a language model for JSON and most of the time it complies. Every so often it wraps the object
Continue readingConstrained Decoding: How LLMs Get Forced Into Valid JSON
In August 2026, ByteDance founder Zhang Yiming told the company’s Seed AI research team something almost nobody in the industry
Continue readingThe Distillation Wars: Which Chinese AI Labs Actually Train Their Own Models
Every token an LLM emits costs a full forward pass. The GPU reads hundreds of gigabytes of weights, computes one
Continue readingSpeculative Decoding: The Math, the Drafters, and When the Bet Pays Off
Every LLM serving stack lives or dies by one data structure most developers never see: the KV cache. It is
Continue readingThe KV Cache: Why LLM Serving Is a Memory Problem
Every modern large language model you use is almost certainly a Mixture-of-Experts model. Mixtral, DeepSeek-V3, GPT-OSS, the latest Kimi and
Continue readingInside Mixture-of-Experts: Routing, Load Balancing, and Why 671B Parameters Cost 37B
Most teams treat “thinking mode” as a switch. Turn it on and the model reasons through the problem step by
Continue readingThinking Inertia: Why LLMs Don’t Stop Reasoning When You Tell Them To
Every RAG pipeline starts with the same piece of code: split documents into chunks, embed them, store them. It looks
Continue readingRAG Chunking Is Not Settled: What Token-Level Evaluations Actually Show