Every token an LLM emits costs a full forward pass. The GPU reads hundreds of gigabytes of weights, computes one row of logits, and hands back a single token — then does it all over again. Arithmetic throughput is not the bottleneck; memory bandwidth is. The GPU spends most of a decode step streaming model weights from HBM, and a single token’s worth of matrix math barely keeps it busy.
Speculative decoding attacks this mismatch with a simple bet: let something cheap guess several tokens ahead, then have the expensive model check all the guesses in one pass. When the guesses land, you get multiple tokens per forward pass at identical output quality. When they miss, you fall back to one token — the same speed you had before. This post walks through the verification math that makes the bet lossless, the trade-offs that decide whether it pays off, and how to actually turn it on in the two inference engines most deployments use.
The core idea: draft fast, verify in parallel
Standard autoregressive decoding is strictly sequential — token t+1 depends on token t, so there is nothing to parallelize during decode. Speculative decoding breaks the dependency in two phases:
- Draft phase: a lightweight drafter — a small language model, a few extra prediction heads, or even an n-gram table — proposes the next K tokens in near-linear time.
- Verify phase: the full target model scores all K draft tokens in a single forward pass. Because attention over a short sequence is cheap relative to weight loading, scoring K tokens costs almost the same as scoring one.
The verifier accepts the longest prefix it agrees with and, on a mismatch, resamples from a corrected distribution. Either way, at least one token emerges per cycle: the verified prefix, plus one freshly sampled token from the final position. In the worst case — every draft rejected — you paid a small draft overhead and got exactly one target-quality token. In the best case, one forward pass yields K+1 tokens.
Why the output is (provably) unchanged
The part that surprises people: accepting or rejecting drafts does not change the output distribution at all. The mechanism is modified rejection sampling. Let p(x) be the target model’s probability for token x and q(x) the drafter’s. A draft token is accepted with probability
min(1, p(x) / q(x))
If the draft token is rejected, the replacement token is sampled from the residual distribution max(0, p(x) − q(x)), renormalized. Working through the algebra, the probability of emitting any token x after this procedure collapses to exactly p(x). If the drafter proposes tokens the target likes, acceptance is high and the residual rarely gets touched; if the drafter guesses poorly, rejection corrects it. Either way the sequence is statistically indistinguishable from what the target model would have produced alone. That result comes from the original speculative sampling papers (Leviathan et al. and Chen et al., both 2023), and it is why the technique is called lossless.
One practical caveat: the guarantee holds for the algorithm as specified. Some serving stacks short-circuit verification under greedy sampling or handle temperature in ways that quietly drop the q(x) term, which turns an ostensibly lossless pipeline into a slightly lossy one. If bit-exact equivalence to the target model matters for your evaluation methodology, verify with the acceptance-rate metrics your engine exposes rather than trusting the label.
The speedup formula and what actually moves it
The expected tokens generated per verification cycle is
E[tokens] = (1 − α^(K+1)) / (1 − α)
where α is the per-token acceptance rate and K the number of drafted tokens. Wall-clock speedup is that quantity multiplied by the ratio of draft cost to target cost, minus the drafter’s own latency. Two failure modes fall straight out of the formula:
- Low α: if acceptance sits near 0.3, deeper drafts mostly waste verification compute — the tree of guesses dies early. The fix is a better drafter, not more draft tokens.
- High K with mediocre α: because α^(K+1) shrinks fast, extra draft length stops paying once acceptance misses compound. Most deployments land on K between 3 and 8, tuned per model.
Acceptance rate is workload-dependent, and this is where practitioners get burned. Code-heavy or structured output (JSON, SQL, boilerplate) drafts beautifully — the next token is often nearly deterministic. Highly creative sampling at high temperature lowers acceptance across the board. And critically, large-batch serving shrinks the win: at batch size 1 the GPU is memory-bound with idle compute, so verification is nearly free. At batch 64 the GPU is compute-bound, verification competes with the main batch, and speculation can even reduce throughput. Speculative decoding is a latency optimization for low-to-medium QPS first, and a throughput optimization only when the drafter is nearly free.
Drafter families: from n-grams to EAGLE-3
The drafter choice determines everything. The main families, in rough order of sophistication:
- Independent draft model. A small LM (say, a 1B drafting for a 70B) from the same tokenizer family. Simple, but acceptance is limited — typically 0.5–0.7 — because the drafter’s context understanding is much weaker than the target’s.
- N-gram / prompt lookup. No model at all: look up continuations from the prompt or recent output in a hash table and propose exact matches. Extremely effective for summarization, code editing, and multi-turn chat where the answer echoes the input. Near-zero draft cost, modest gains, and it adds no VRAM pressure at peak traffic.
- Medusa. Extra decoding heads attached to the target model that predict several future tokens per position in parallel. No separate model to serve, but standalone acceptance per head is lower (~0.5–0.65); it relies on tree-structured candidate combinations.
- EAGLE (1/2/3). Drafts at the feature level — it predicts the target model’s hidden state one step ahead rather than raw tokens, reusing the target’s own top-layer features. EAGLE-2 adds context- and draft-aware tree expansion; EAGLE-3 abandons feature prediction for direct token prediction over a multi-layer feature fusion trained with a “training-time test” scheme. On standard benchmarks EAGLE-3 pushes acceptance into the 0.8–0.9 range and delivers speedups up to 6.5× at batch size 1, with the SGLang documentation reporting 373 tokens/s versus 158 baseline (2.4×) on Llama-3.1-8B with a single H100. The paper also reports 1.38× throughput improvement at batch size 64 in SGLang — evidence that a strong enough drafter can keep paying even under batching.
- MTP (multi-token prediction). Models trained natively with multiple prediction heads (the DeepSeek lineage) ship a built-in drafter. When your target model has MTP weights, this is usually the lowest-friction high-performance option.
Turning it on in vLLM
vLLM exposes speculation through a single JSON config on the CLI (the same keys work via the Python LLM(speculative_config=...) API). A typical EAGLE setup:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--speculative-config '{
"method": "eagle",
"model": "yuhuili/EAGLE3-Llama-3.1-8B-Instruct",
"num_speculative_tokens": 5
}'For models that ship native MTP weights, the method becomes "mtp" and the draft model path is often unnecessary. N-gram speculation needs no model at all:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--speculative-config '{
"method": "ngram",
"num_speculative_tokens": 5,
"prompt_lookup_num_tokens": 10
}'vLLM’s own method selection table gives honest guidance on where each variant wins: model-based drafters (EAGLE, MTP) take the top slot for low-QPS latency, while n-gram and suffix decoding provide “modest speedups without increasing workload during peak traffic” — a trade worth remembering when your cluster is saturated.
Turning it on in SGLang
SGLang’s implementation is flag-based and generally regarded as among the fastest open-source paths for EAGLE-3:
python -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--speculative-algorithm EAGLE3 \
--speculative-draft-model-path yuhuili/EAGLE3-Llama-3.1-8B-Instruct \
--speculative-num-steps 5 \
--speculative-eagle-topk 4 \
--speculative-num-draft-tokens 8The three tuning knobs interact: --speculative-num-steps is the drafting depth, --speculative-eagle-topk the branching factor per step, and their product bounds the draft tree size (--speculative-num-draft-tokens). Wider trees raise acceptance but consume more verification compute and KV cache — the same α-versus-K tension from the formula, now with VRAM consequences. SGLang’s docs ship a full parameter reference and an OOM troubleshooting section precisely because aggressive speculation settings are the most common way operators crash a otherwise-healthy server.
Operational checklist
- Measure acceptance, not vibes. Both engines expose per-request and aggregate acceptance metrics. If acceptance sits below ~0.5 with a model-based drafter, the drafter is mismatched to your workload — retrain, switch families, or disable.
- Benchmark at your real batch sizes. A config that doubles single-stream latency can be neutral or negative at batch 32. Test with production-shaped traffic (vLLM’s benchmark CLI and SGLang’s bench_serving both support concurrency sweeps).
- Mind the tokenizer. Classic verification requires draft and target to share a vocabulary. Cross-tokenizer speculation exists but needs string-level verification; don’t assume a great small model is a valid drafter.
- Watch peak memory. Draft trees multiply effective batch size during verification. Enable speculation only after confirming headroom, and check the OOM guidance before pushing topk above 4–5.
- Greedy parity check. If you rely on deterministic outputs for evals, run your eval suite with and without speculation and diff. A correct implementation is bit-identical under greedy decoding; divergence signals a shortcut in the sampling path.
Speculative decoding has quietly become a default rather than an option — the current-generation reasoning and chat models almost all ship with MTP heads or an official EAGLE-3 drafter, and both major serving engines enable it with one flag. The math guarantees you never ship worse tokens; the only decision left is whether your traffic pattern lets the bet pay. Measure acceptance at realistic concurrency, tune K down before you tune it up, and keep the zero-cost n-gram path in your back pocket for batch-bound peaks.