Large language model inference has an awkward performance profile: the GPU does enormous math, then waits. Every token requires a full forward pass through the model, and because each token depends on the one before it, there is no way to parallelize the sequence at generation time. At low and medium query volumes the bottleneck is not compute at all — it is memory bandwidth. The model weights are read from HBM to produce a single token, and then read again for the next one. Speculative decoding attacks exactly this inefficiency, and vLLM now ships one of the most complete implementations of it anywhere.
The idea sounds suspiciously like getting something for nothing: run a small, cheap model to draft several tokens, then have the large model verify all of them in a single forward pass. If most of the drafts survive verification, you have decoded several tokens for the cost of one round trip through the big model. The reason this is not a trick with a hidden quality cost is that verification uses rejection sampling — the draft tokens are only accepted if the target model would have produced them anyway, and the correction step repairs any divergence. The output distribution is preserved, which is why the technique is described as lossless. The original speculative sampling paper works through the proof, but the practical summary is: with greedy decoding, output with speculation matches output without it, token for token — up to the limits of floating-point precision, which can still cause rare divergences on real hardware.
Why Generation Is Memory-Bound
A useful mental model: decoding an 8B-parameter model at batch size one reads roughly 16 gigabytes of weights (at bf16) to produce each token. The actual matrix arithmetic on a single token vector is trivial; almost all the time is spent streaming weights through memory. Now add a draft model that is fifty times smaller. Reading the small model’s weights five times and the big model’s weights once still costs far less bandwidth than reading the big model’s weights five times. That arithmetic is the whole bet — and when the drafter guesses well, the effective speedup at low concurrency routinely lands between 1.5x and 3x depending on the method and workload.
The Method Menu
vLLM’s speculative decoding documentation has grown from a single draft-model path into a menu of proposers, each with a different profile:
- Draft model — a second, smaller model from the same family proposes tokens. The classic approach; needs a compatible checkpoint and works best when a good small sibling exists.
- EAGLE — instead of a separate model, a lightweight head attached to the target model predicts tokens from the target’s own hidden states. Strong general-purpose gains at both low and high load. The EAGLE guide lists compatible checkpoints.
- Multi-Token Prediction (MTP) — uses MTP weights trained alongside the target model, where available. When the model family ships native MTP modules, this is usually the best-supported path; see the MTP guide for which families qualify.
- N-gram — no second model at all. The proposer looks for matching n-grams between the prompt and the generated text so far, which makes it nearly free and surprisingly effective for tasks with repetitive structure: code editing, summarization, RAG answers that echo retrieved chunks.
- Suffix decoding — builds a global suffix tree across requests and speculates from long prefix matches. Originally aimed at agentic workloads where the same tool-call scaffolding repeats; the speculation depth adapts to match length.
Choosing between them is mostly a question of where you sit on the latency-throughput curve. Model-based methods (EAGLE, MTP, draft models) buy the largest latency reductions when concurrency is low and the GPU is starved. As QPS climbs and batches fill up, the GPU becomes compute-bound, speculative work starts competing with real requests for FLOPs, and the lighter n-gram and suffix methods — which add almost no overhead — become the safer default. There is also a dynamic mode that adjusts speculation behavior as load fluctuates, aimed at RL rollouts and other bursty workloads.
Configuration on the Serve Path
Everything routes through a single JSON object passed to --speculative-config on the CLI, or the speculative_config dict in the Python API. A same-family draft setup looks like this:
vllm serve Qwen/Qwen3-8B \
--speculative-config '{
"method": "draft_model",
"model": "Qwen/Qwen3-0.6B",
"num_speculative_tokens": 5
}'The n-gram path needs no second checkpoint at all, just lookup window bounds:
vllm serve Qwen/Qwen3-8B \
--speculative-config '{
"method": "ngram",
"num_speculative_tokens": 4,
"prompt_lookup_min": 2,
"prompt_lookup_max": 5
}'The offline Python API takes the same keys. Historically the draft and target had to share a tokenizer, which locked you into same-family pairs. That constraint is now relaxable: enabling use_heterogeneous_vocab turns on the Token-Level Intersection algorithm, which maps the two vocabularies at startup, constrains the drafter to shared tokens, and translates token IDs before verification. That means a tiny general-purpose drafter can accelerate a model from a different family entirely:
from vllm import LLM
llm = LLM(
model="Qwen/Qwen3-8B",
speculative_config={
"method": "draft_model",
"model": "HuggingFaceTB/SmolLM2-135M-Instruct",
"num_speculative_tokens": 3,
"use_heterogeneous_vocab": True,
},
gpu_memory_utilization=0.5,
)One caveat worth knowing before you lean on this: cross-vocabulary drafting currently supports greedy draft sampling only — probabilistic acceptance with a temperature above zero on the drafter is not yet implemented. And if your drafter needs its own parallelism, the key is draft_tensor_parallel_size; plain tensor_parallel_size inside the speculative config is rejected.
Tuning Depth, and When Speculation Backfires
num_speculative_tokens is the knob everyone reaches for first, and its optimal value is entirely workload-dependent. High acceptance rates reward deeper speculation; creative, high-temperature generation produces drafts the target rejects quickly, and every rejected token is wasted compute on both models. The failure mode to watch for is not corruption — the output stays correct — it is throughput collapse at high load, where speculative work crowds out paying requests. Pipeline parallelism is also not composable with speculative decoding, which matters for large multi-node deployments.
Because acceptance rate is the health metric for the whole feature, vLLM exposes per-request acceptance metrics you can log and alert on. A deployment that accepted 80 percent of drafted tokens last month and 40 percent this month has a changed workload or a drifted drafter, and that belongs in your observability, not in a quarterly benchmark.
Measure Before You Ship
Every speedup claim in this area is an average over someone else’s workload. The honest numbers come from running the reproducible benchmark in the vLLM repo — examples/features/speculative_decoding/spec_decode_offline.py — or the benchmark CLI against your own traffic mix. Test at your real concurrency, with your real sampling parameters, and compare both time-to-first-token and inter-token latency.
If you control the model side too, the speculators project lets you train draft heads specifically for your target model and distribution, which tends to beat generic drafters by a comfortable margin. For everyone else, the pragmatic ladder is: try n-gram first (it is nearly free), move to EAGLE or MTP if your model family supports it, and reserve cross-vocabulary drafting for cases where no same-family drafter exists. The vLLM repository documents the compatibility matrix per release — it moves fast enough that yesterday’s limitation is often this week’s changelog entry.
Speculative decoding is one of the rare optimizations that does not ask you to trade output quality for speed. The math guarantees the distribution; your job is just to check whether your workload’s predictability — and your concurrency level — makes the draft-verify bet worth taking.