Autoregressive decoding is the reason large language models feel slow: every token is generated by a full forward pass, and generating a thousand-token response means a thousand sequential passes. Speculative decoding attacks this directly — a small draft model guesses several tokens ahead, and the large target model verifies them all in a single pass. Same output distribution, far fewer forward passes. When it works, you get 2–3x lower latency for free.
When it doesn’t work, it’s usually because the draft model is a bad guesser. This post covers how speculative decoding works, why naive draft models cap out quickly, and how EAGLE-style methods fix the problem by predicting at the feature level instead of the token level.
The mechanics: draft, verify, accept
The loop has three steps:
- Draft: a cheap draft model autoregressively generates k candidate tokens (typically 3–8).
- Verify: the target model runs one forward pass over the original prompt plus all k draft tokens. Because the pass processes all positions in parallel, verifying k tokens costs roughly the same as generating one.
- Accept: walk the draft tokens left to right, comparing the draft’s proposed tokens against the target model’s output distribution. A draft token is accepted if the target would have produced it (exactly, under greedy decoding); on the first mismatch, reject that token and everything after it, and continue from the target’s own choice.
With a rejection-sampling correction (the standard formulation accepts a draft token with probability min(1, p_target/p_draft) and resamples on rejection), the output distribution is provably identical to what the target model would produce on its own. This is not lossy compression of the output — it’s a latency optimization with a correctness guarantee. In greedy mode the guarantee is exact: the sequence produced is exactly the greedy sequence of the target model.
The expected speedup follows a simple rule of thumb: if the draft is accepted with probability α and you draft k tokens, you expect roughly (1 − α^(k+1))/(1 − α) tokens per verify pass, at the cost of the draft model’s own runtime. High α is everything — which is where most implementations fall apart.
Why naive draft models underdeliver
The obvious approach — take a small model (say, a 1B-parameter model to draft for a 70B target) and run the loop — hits two problems:
Acceptance rates are mediocre. A small model with a different tokenizer, vocabulary, or training distribution diverges from the target quickly. Typical acceptance rates for independent small-model drafters land around 0.6–0.8; combined with the draft model’s own decode cost per token, the net speedup shrinks toward 1.5x or less.
Alignment is hard to maintain. The draft and target must agree not just on what token comes next, but on the model’s internal reasoning state. A model trained on different data at a different scale develops different internal representations, so early divergence compounds over the drafted horizon.
EAGLE: draft at the feature level, not the token level
EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) reframed the problem. Instead of a separate small language model, EAGLE uses a tiny autoregressive head attached to the target model that operates on the target’s top-layer feature vectors — the hidden states just before the output projection — rather than on token IDs alone. The draft head is trained with the target model frozen, is a small fraction of the target’s size, and reuses the target’s embedding layers, so there is no vocabulary or tokenizer mismatch to manage.
Because the draft conditions on the target’s own internal features, its predictions track the target’s reasoning much more closely. The original EAGLE reported roughly 3x speedup over vanilla decoding, and EAGLE-2 improved the draft tree construction with context-aware dynamic drafting to push further. The limits of this line became clear, though: scaling the draft head’s training data yielded diminishing returns, because feature prediction itself was the bottleneck.
EAGLE-3: training-time test and multi-layer fusion
EAGLE-3 made two changes that remove that ceiling:
- Multi-layer feature fusion. Instead of consuming only the top layer’s features, the draft head fuses features from several lower, middle, and upper layers. Low-level layers capture syntax and local structure; high-level layers capture semantics. The fused representation gives the draft head more signal about the target’s state than any single layer.
- Training-time test. During training, the draft head is exposed to inputs that simulate inference-time conditions — the model sees its own drafted (not ground-truth) prefixes, in the way it will actually be queried at deployment. This closes the train/inference distribution gap that previously capped how much extra training data could help. With the gap closed, scaling training data finally translated into better acceptance rates.
EAGLE-3 also abandons pure feature prediction in favor of direct token prediction on top of the fused features. The reported results: speedup ratios up to 6.5x over vanilla decoding — about 1.4x better than EAGLE-2 — and a 1.38x throughput improvement at batch size 64 in SGLang. Notably, the gains hold for reasoning models, not just chat models, which matters as chain-of-thought-style outputs dominate real traffic (longer outputs mean more drafted positions, so speculative decoding’s payoff scales with response length).
Running it: vLLM and SGLang
Both major serving stacks ship EAGLE-3 support. In vLLM, enable it with --speculative-config, pointing at a pretrained EAGLE-3 draft head for your target model:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--speculative-config '{"method": "eagle3", "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B", "num_speculative_tokens": 5}'In SGLang, the flag is --speculative-algorithm EAGLE3. Community-maintained EAGLE-3 draft heads are available on Hugging Face for the common open-weight families (Llama, Qwen, Vicuna, and others) under the yuhuili organization, and the reference implementation with official checkpoint links lives in the SafeAILab/EAGLE repository.
The number that matters in production is the draft acceptance rate — in vLLM, the spec_decode_draft_acceptance_rate metric. Below roughly 0.7, the draft overhead starts eating the wins; above 0.8 with 4–6 drafted tokens, 2–3x speedups are typical. The other tunable is the number of speculative tokens: more tokens raise the ceiling of tokens per verify pass but waste more draft compute on later, less certain positions. Tune it against your traffic — batch size matters a lot here, since under heavy load the verify pass is no longer cheap relative to draft compute, and the effective speedup compresses.
When speculative decoding is and isn’t worth it
Speculative decoding shines when latency dominates: interactive chat, agent loops, code completion — anywhere a user is waiting on tokens. It shines less when throughput at high concurrency is the goal (draft compute competes with batch capacity) or when the response is dominated by prefill rather than decode (summarizing a long document, for instance, is prefill-bound and gains little).
The EAGLE line changed the calculus by making acceptance rates high enough that the technique stopped being a research demo and became a default serving configuration. If you’re serving an open-weight model today and haven’t tried attaching a draft head, you’re likely paying 2–3x more latency than you need to.