Every modern large language model you use is almost certainly a Mixture-of-Experts model. Mixtral, DeepSeek-V3, GPT-OSS, the latest Kimi and
Continue readingInside Mixture-of-Experts: Routing, Load Balancing, and Why 671B Parameters Cost 37B
Most teams treat “thinking mode” as a switch. Turn it on and the model reasons through the problem step by
Continue readingThinking Inertia: Why LLMs Don’t Stop Reasoning When You Tell Them To
Ask someone to sketch a transformer on a whiteboard and they’ll draw the 2017 version: softmax attention over every pair
Continue readingMHA, MQA, GQA, MLA: The KV-Cache Trade-offs Behind Every Modern Transformer
Run two unrelated documents through a large language model at the same time, and you assume the model keeps them
Continue readingYour Transformer Can Hold Two Thoughts at Once: The New Linear Superposition Result