Every modern large language model you use is almost certainly a Mixture-of-Experts model. Mixtral, DeepSeek-V3, GPT-OSS, the latest Kimi and Qwen releases — nearly every frontier-scale open-weight model of the past two years has adopted MoE as its core architecture. Yet most explanations stop at the elevator pitch: “instead of one big feed-forward network, there are many expert networks and a router picks a few.” That pitch hides all the engineering. Routing is not a simple argmax, experts do not stay balanced by accident, and the “sparse” model still has to hold every parameter in memory. This post walks through how MoE layers actually work, why the router is the hardest part of training one, and what the numbers look like for real production models.
By the end you should be able to read a model card like “671B total parameters, 37B activated” and understand precisely what it means — and why that gap between total and active parameters is both the entire point of the architecture and the source of most of its problems.
The architecture: swap one FFN for many
A standard transformer block is attention followed by a feed-forward network (FFN). The FFN holds roughly two-thirds of a dense model’s parameters, and every token passes through all of them. The MoE idea, which goes back to a 1991 paper by Jacobs and Jordan and was revived for transformers in the GShard line of work, is simple: replace that single FFN with a set of N expert FFNs plus a small gating network called the router. For each token, the router scores all experts and sends the token to only the top-k of them. The rest of the parameters sit idle for that token.
A concrete example makes the sparsity tangible. Mixtral 8x7B has 8 experts per layer and routes each token to 2 of them. Each expert is a standard FFN at the model’s hidden size, and the router itself is just a linear projection from the hidden dimension down to 8 logits followed by a softmax. DeepSeek-V3 scales the pattern up: its config defines 256 routed experts with 8 selected per token, plus one “shared” expert that every token passes through unconditionally. GPT-OSS-120B sits in between with 128 experts, 4 active per token.
The routing math is worth spelling out, because almost everything that follows is a consequence of it:
import torch
import torch.nn.functional as F
def route(x, W_g, top_k):
# x: (batch, seq_len, hidden)
# W_g: (hidden, num_experts) - the router is a single linear layer
logits = x @ W_g # (batch, seq_len, num_experts)
probs = F.softmax(logits, dim=-1)
top_p, top_i = torch.topk(probs, top_k, dim=-1)
top_p = top_p / top_p.sum(dim=-1, keepdim=True) # renormalize over selected experts
return top_i, top_p # which experts, and how to weight themThe token’s final representation is the weighted sum of its selected experts’ outputs. Note the two design choices that vary across model families: whether probabilities are renormalized over just the selected experts (Mixtral does; some earlier designs skip it), and whether the softmax is over all experts up front (the alternative, used by DeepSeek-V3 among others, is a sigmoid per expert followed by normalization, which decouples each expert’s score from the others).
Total vs. active parameters
Here is what the model-card numbers actually decompose into. Attention layers, embeddings, and the router are shared and dense — every token touches them. Only the expert FFNs are sparse. So total parameter count is dominated by experts (N copies of the FFN per layer), while per-token compute cost tracks the active subset (k copies) plus the dense parts.
DeepSeek-V3 is the cleanest illustration: 671B total parameters, 37B activated per token. That ratio is not a rounding artifact — it is the design. Compute cost is what it would be for a ~37B dense model, but capacity (what the model can in principle know) is closer to a 671B one. You get the knowledge capacity of a huge model at the inference cost of a modest one.
The catch is that memory does not follow compute. To route a token, every expert must be resident and reachable, so serving a 671B/37B MoE still requires hardware that holds all 671B parameters. That is why MoE models are famously awkward to self-host: the quantized checkpoints that make dense models fit on a single GPU do not change the fact that MoE models hold 4-25x more parameters than they activate per token, depending on the architecture. The trade is “cheap per token, expensive per deployment.”
The router is the whole ballgame
If the architecture is that simple, why do MoE papers spend most of their pages on routing? Because the router is trained by gradient descent and will happily destroy itself. Two failure modes dominate.
Failure 1: routing collapse
Early in training, one or two experts pull ahead — random initialization gives them slightly better outputs, so the router sends them more tokens, so they get more gradient signal, so they improve faster, and the rich-get-richer loop runs away. Soon the model is a dense model wearing a MoE costume: all tokens funnel to the same handful of experts while the rest never train. This is called routing collapse, and it is the default outcome, not the exception, in unregularized MoE training.
Failure 2: expert capacity overflow
Expert FFNs process fixed-shape tensors on hardware that hates ragged shapes. So implementations assign each expert a capacity: a maximum number of tokens per batch, typically computed as (tokens in batch / number of experts) times a capacity factor around 1.0-1.5. Tokens routed to an expert whose buffer is already full are dropped — they skip the FFN entirely and pass through via their residual connection. Under-trained routers drop a lot of tokens; the ST-MoE paper found that fine-tuning quality is not materially impacted by dropping up to 10-15% of tokens, but chronic overflow means some inputs are systematically getting less model than others.
The standard fix: auxiliary load-balancing loss
The classic remedy from the Switch Transformer paper is to add a second loss term that penalizes imbalance. For each MoE layer, compute f_i, the fraction of tokens routed to expert i, and P_i, the average router probability assigned to expert i. The auxiliary loss is the sum of f_i times P_i across experts, scaled by a small coefficient alpha (typically around 0.01). If routing were perfectly uniform, every f_i and P_i would equal 1/N and the loss would hit its minimum of 1/N. Pushing this term down forces the router to spread both traffic and confidence evenly.
The problem with the auxiliary loss is that it fights the language-modeling loss. Uniformity is not the same as quality — sometimes a token genuinely needs a particular expert, and the balancing term taxes that. The coefficient becomes a tuning tightrope: too small and routing collapses anyway; too large and model quality degrades measurably because the router optimizes for fairness over correctness.
The modern fix: balance without gradients
DeepSeek-V3 took a different route, and its README describes it plainly: an “auxiliary-loss-free strategy for load balancing, which minimizes the performance degradation that arises from encouraging load balancing.” The mechanism is a per-expert bias added to the router’s affinity scores only at the top-k selection step — never to the weighting values used to combine expert outputs. After each step, the bias of overloaded experts is decremented and underloaded ones incremented. Because the bias never enters the gradient path, the router learns accurate preferences while the bias term alone handles traffic shaping. There is still a small complementary sequence-wise balance loss, but the dominant balancing signal is now gradient-free.
This design has been widely adopted in the newer generation of open-weight MoE models, and it resolves the tightrope problem: load balance stops being something you buy with model quality.
A minimal MoE layer in PyTorch
Tying it together, here is a compact but complete top-k MoE layer with the bias-based balancing hook, mirroring how modern implementations structure it:
import torch
import torch.nn as nn
import torch.nn.functional as F
class MoELayer(nn.Module):
def __init__(self, hidden, num_experts, top_k, capacity_factor=1.25):
super().__init__()
self.router = nn.Linear(hidden, num_experts, bias=False)
self.experts = nn.ModuleList(
nn.Sequential(nn.Linear(hidden, 4 * hidden), nn.GELU(),
nn.Linear(4 * hidden, hidden))
for _ in range(num_experts)
)
self.register_buffer("expert_bias", torch.zeros(num_experts))
self.top_k = top_k
self.capacity_factor = capacity_factor
def forward(self, x):
B, S, H = x.shape
x_flat = x.reshape(-1, H) # (B*S, H)
scores = self.router(x_flat) # logits, no softmax yet
scores_for_pick = scores + self.expert_bias # bias affects SELECTION only
top_p, top_i = torch.topk(scores_for_pick, self.top_k, dim=-1)
# combine weights come from the UNBIASED scores
weights = torch.softmax(scores.gather(1, top_i), dim=-1)
num_tokens = x_flat.size(0)
cap = int(self.capacity_factor * num_tokens * self.top_k / len(self.experts))
out = torch.zeros_like(x_flat)
dropped = 0
for e, expert in enumerate(self.experts):
tok_i, slot = (top_i == e).nonzero(as_tuple=True)
if tok_i.numel() == 0:
continue
if tok_i.numel() > cap: # capacity overflow: drop
dropped += tok_i.numel() - cap
tok_i, slot = tok_i[:cap], slot[:cap]
out.index_add_(0, tok_i, expert(x_flat[tok_i]) * weights[tok_i, slot].unsqueeze(1))
self.last_drop_rate = dropped / max(1, num_tokens * self.top_k)
return out.reshape(B, S, H)This is a training-shaped reference implementation, not a serving kernel — real engines replace the per-expert loop with grouped GEMMs and all-to-all collective communication across GPUs (expert parallelism shards experts, not tokens, across devices, and tokens get routed over the network between them). But the structure is faithful: unbiased scores for weighting, biased scores for selection, a capacity limit with real dropping, and a bias buffer that a training loop would adjust after each step.
What this means when you read a model card
A few practical takeaways for anyone evaluating or deploying these models:
- Active parameters are an inference-cost estimate, not a quality score. Two models with the same active count can differ wildly in total capacity, and frontier MoE models deliberately exploit that gap.
- Memory planning must use the total count. If you are self-hosting, VRAM requirements track the full parameter set (plus KV cache); the “37B activated” number is irrelevant to your GPU shopping.
- Routing is per token, not per request. Different tokens in the same prompt go to different experts. There is no single “expert for coding” — interpretability work suggests experts specialize on token-level patterns far more than on topics.
- Quantization interacts unevenly with experts. Router weights are tiny but precision-sensitive; quantizing the router itself can flip expert selections and degrade output noticeably even when the expert FFNs quantize cleanly.
- Batch size matters more than for dense models. MoE efficiency gains assume enough concurrent tokens to keep every expert’s buffer usefully full. At very low batch sizes, expert parallelism overhead can erase the theoretical advantage.
Wrapping up
Mixture-of-Experts is best understood not as one architecture but as a bargain: pay memory for many experts, run compute for only a few, and solve the resulting traffic-management problem in the router. The last three years of model releases show the bargain maturing — top-k routing with renormalized gates, capacity factors with token dropping, and now gradient-free load balancing that no longer taxes model quality. If you want to go deeper, the Switch Transformer and DeepSeek-V3 papers bracket the evolution well: read the first to understand why balancing losses exist, and the second to see where the field has moved. And if you self-host models, the next post in this series on inference scheduling covers what MoE sparsity does to batching decisions in serving engines like vLLM.