Generating text from a large language model is an inherently serial process. The model produces one token, appends it to the sequence, and produces the next. Each step requires a full forward pass over the network, and the arithmetic intensity of that pass is so low at batch size 1 that the GPU spends most of its time streaming weights from memory rather than computing. You buy an accelerator rated for tens of thousands of tokens per second and watch it produce thirty.
Speculative decoding attacks this bottleneck from an unexpected angle. Instead of making the big model faster, it lets a cheap proposer guess several tokens ahead, then makes the big model check all of those guesses in a single pass. The remarkable part is not the speedup — typically 1.4x to 3x in practice — but the guarantee that comes with it: done correctly, the output distribution is mathematically identical to what the big model would have produced on its own. You get shorter wall-clock times without changing a single sampled token.
Draft, Then Verify
The core loop has two roles. A draft model — a much smaller network, or even a heuristic — autoregressively proposes a short continuation of K tokens. The target model, the one you actually wanted to run, then scores the entire sequence prompt-plus-drafts in one forward pass. Because the pass is computed in parallel over positions, verifying K tokens costs roughly the same as generating one: the dominant expense is reading the weights, and that happens once either way.
After verification, the loop compares each draft token against what the target model would have chosen. Accepted tokens are kept; the first rejection is replaced by the target model’s own choice, and everything after it is discarded because the draft reasoned from a token that no longer exists. Even a fully rejected draft costs little — the target model still emits exactly one token per step, so the worst case is a slowdown proportional to draft cost, never a wrong answer.
The Guarantee Behind Rejection Sampling
Keeping draft tokens verbatim would bias the output toward the draft model’s opinions. The fix is a modified rejection test, described independently by teams at Google and DeepMind. For each drafted token, the token is accepted with probability min(1, p(t)/q(t)), where p is the target model’s probability and q the draft model’s. If the draft was more confident than the target about a token, that token is accepted outright; if less, it survives only probabilistically, and on rejection a residual distribution is used to resample a corrected token.
The consequence is provable: the resulting sequence is distributed exactly as if the target model had sampled every token itself. When you run with greedy decoding the test collapses to a simple check — keep the draft token only if it matches the target’s argmax, otherwise take the argmax — which is what most production systems do because it preserves their deterministic outputs.
How many tokens survive? If α is the probability that a single draft token is accepted and the draft proposes K tokens, the expected number of tokens emitted per verification pass approaches (1 − α^(K+1)) / (1 − α). With α around 0.7 and K around 4, that is roughly 2.9 tokens for the price of one target-model pass. Real-world acceptance rates tend to land in the 0.6–0.8 range for well-matched draft models, which is where the commonly observed 1.4x–3x speedups come from.
Seeing It in Code
The acceptance test is a few lines. Here it is written out explicitly, for the sampling case:
import random
def accept_draft_token(p_target: dict, q_draft: dict, token: str) -> tuple[bool, dict]:
"""One step of the modified rejection test.
Returns (accepted, residual) where residual is the distribution to
sample a replacement token from if the draft token is rejected.
"""
p = p_target.get(token, 0.0)
q = q_draft.get(token, 1e-9)
ratio = p / q
if ratio >= 1.0:
return True, {}
if random.random() < ratio:
return True, {}
# Residual: normalized max(0, p - q) — the correction that makes the
# combined output distribution identical to the target model's.
residual = {t: max(0.0, p_target[t] - q_draft.get(t, 0.0))
for t in p_target}
total = sum(residual.values())
if total > 0:
residual = {t: v / total for t, v in residual.items()}
return False, residualAt the verification level, the flow looks like this — draft K candidates, score them all against the target’s distribution in one pass, and walk the list until the first disagreement:
def speculative_step(target, draft, tokens: list[int], k: int = 4) -> list[int]:
# 1. Draft: k cheap autoregressive proposals.
candidates = draft.propose(tokens, k=k)
# 2. Verify: ONE target-model forward pass over tokens + candidates.
probs = target.score_all_positions(tokens + candidates)
# 3. Walk the draft tokens; stop at the first mismatch.
emitted = []
for i, cand in enumerate(candidates):
if cand == argmax(probs[i]):
emitted.append(cand) # draft was right — keep it
else:
emitted.append(argmax(probs[i])) # replace with target's choice
break # everything after is invalid
return emittedIn practice you do not write this loop yourself. The transformers generation API exposes it directly: pass a smaller model as assistant_model and the same verification machinery runs inside generate().
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "meta-llama/Llama-3.1-8B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
target = AutoModelForCausalLM.from_pretrained(name, device_map="auto")
draft = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-1B-Instruct", device_map="auto"
)
inputs = tok("Summarize how a write-ahead log protects a database:",
return_tensors="pt").to(target.device)
out = target.generate(**inputs, assistant_model=draft, max_new_tokens=256)
print(tok.decode(out[0], skip_special_tokens=True))Note the pairing: Llama 3.2 1B shares its tokenizer with Llama 3.1 8B, which matters — if the draft model segments text differently, draft tokens cannot be scored by the target at all. On the serving side, vLLM enables the same technique with a JSON config, including proposers that need no second model at all:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--speculative-config '{
"method": "ngram",
"num_speculative_tokens": 4,
"prompt_lookup_min": 2,
"prompt_lookup_max": 5
}'The ngram proposer scans the prompt and already-generated text for repeating sequences and uses the continuation as its draft. For workloads with heavy copying — summarization, code editing, structured output echoing the input — it is surprisingly effective and costs no GPU memory for a second model.
Beyond Two Models: The EAGLE Line
Using a separately trained small model as the drafter leaves accuracy on the table, because the draft never sees the target model’s internal state. The EAGLE family takes a different approach: a lightweight head autoregressively predicts in the target model’s feature space — one layer before the output — reusing the target’s own embeddings and attention states. Because drafts condition on the target’s actual representations, acceptance rates stay high even deeper into the draft chain.
EAGLE also drafts trees of candidate continuations rather than single chains, and the target verifies all branches in parallel with a tree-shaped attention mask, keeping the branch that scores best. EAGLE-3 adds what its authors call training-time test scaling: training against multiple approximation levels so that acceptance rates hold up as draft depth grows, instead of decaying the way earlier versions did. These drafters are where the upper end of the speedup range comes from; plain draft-model setups more commonly land near the lower end.
Where Speculative Decoding Disappoints
The technique is a latency optimization, and it shines exactly when a decode pass is memory-bandwidth-bound: single requests, small batches, latency-sensitive interactive traffic. Push the batch size up and the picture changes. Verification extends the sequence length by K positions, which consumes compute proportionally, and under heavy load a server is compute-bound rather than bandwidth-bound. At high concurrency, speculative decoding can reduce throughput — that is why serious serving stacks expose it as a per-deployment tuning choice, not a universal default.
A second failure mode is a quality gap between drafter and target. A draft model that is too small proposes tokens the target would never pick; acceptance collapses and you pay draft cost for nothing. The draft must be well matched to both the domain and the tokenizer, and tuned with the same care you would give the target model.
Finally, rejected drafts have KV-cache consequences. The target model computed keys and values for draft positions that no longer belong in the sequence, so the serving engine has to rewind or mask that state. Mature engines handle this internally, but it is one more reason speculative decoding behaves differently depending on the serving stack underneath it.
When to Turn It On
Start with the workload profile. Interactive chat, single-user agents, and code completion — anything where one user waits on one stream — are the target case. Offline bulk generation at high batch parallelism usually is not. Then choose a proposer: n-gram matching first, since it is free; a matched small draft model second; a feature-level drafter like EAGLE if the speedup still matters enough to justify training one. Measure the acceptance rate in production, not just the end-to-end speedup — a falling acceptance rate is an early warning that your traffic distribution has drifted away from whatever the drafter was tuned for.
The deeper lesson is architectural. Decode speed is not a property of the model alone but of the loop around it: what gets proposed, what gets verified, and what gets thrown away. Speculative decoding trades idle memory bandwidth for useful work, and unlike quantization or distillation, it does so without touching the model’s output distribution at all.