Watch a GPU while a large language model generates text and you’ll see something strange: for most of every forward pass, the hardware is barely doing the work it was built for. Autoregressive decoding produces exactly one token per pass, and each pass is dominated by streaming billions of weights from memory. The arithmetic units spend most of that time waiting. This is why LLM decoding is described as memory-bandwidth-bound, and it’s the gap that speculative decoding attacks — not by making the model faster, but by getting more useful work out of each pass.
The core idea is deceptively simple. A cheap draft process guesses several future tokens. The expensive target model then verifies all of them in a single forward pass — something a standard transformer can do, because one pass over k tokens costs roughly the same as one pass over one token when the sequence fits in cache. Accepted guesses carry forward; rejected ones are discarded and recomputed correctly. The result, when it works, is two to three times faster decoding with mathematically identical output to what the target model would have produced alone. That last part is what separates speculative decoding from approximation tricks: rejection sampling guarantees the final token distribution is unchanged.
Why Decoding Leaves the GPU Idle
A decoder-only transformer generates tokens one at a time. At each step, the model runs a full forward pass — every layer, every weight matrix — just to produce the next-token distribution. Prefill, the processing of your prompt, is different: it processes all input tokens in parallel and saturates the GPU nicely. Decode is the slow, lonely phase where batch size is effectively one per request and the bottleneck shifts from compute to the memory bus.
This is also where the KV cache lives. Each layer stores the keys and values of every previous token so attention doesn’t recompute them. Per token, that’s roughly 2 × layers × hidden_size × precision_bytes of memory, multiplied by sequence length and batch size. A 7B model at half precision carries a couple of gigabytes of KV state for a single 4K-token sequence. The practical consequence: during decode, the GPU reads weights and cache state to produce one token, and its compute capacity sits mostly unused. Speculative decoding exploits exactly that spare capacity.
The Speculative Contract
The algorithm has three repeating phases:
- Draft: a fast process proposes the next k tokens autoregressively (or in parallel, depending on the method).
- Verify: the target model runs one forward pass over the drafted tokens, producing its own distribution for each position simultaneously.
- Accept or rewind: tokens are accepted or rejected using a rejection-sampling rule that compares draft and target probabilities. On the first rejection, everything after it is discarded, and the target’s own sample at that position becomes the next token. At least one token is always produced per cycle — the target’s own.
The acceptance rule is the subtle part. It’s constructed so that the probability of any accepted token equals exactly what the target model would have assigned. Speedup therefore depends entirely on the acceptance rate, which is a function of how well the draft approximates the target on your actual workload. Long, predictable spans — code identifiers, boilerplate, template text — accept almost everything. Creative or highly stochastic output accepts less, and each rejected position wastes draft work.
There’s also a throughput tax. Verification passes over k drafted tokens per cycle, so each round costs more FLOPs than plain decoding. When the server is busy — high batch size, KV cache under pressure — those spare cycles aren’t spare anymore. Speculative decoding is fundamentally a latency optimization that trades spare compute for lower time-per-token, and it delivers the most when the GPU is underutilized.
Draft Candidates — Side Models to Self-Drafting
The original formulation uses a small, independent draft model — think a 1B-parameter sibling of the 70B target. The catch is alignment: the draft must share the target’s tokenizer and vocabulary, or verification becomes incoherent. In practice, that usually means a model from the same family, which limits off-the-shelf options.
Newer methods avoid the second model entirely. Medusa trains additional decoding heads on the target model that predict several future tokens in parallel; a tree-attention step verifies the most promising combinations. EAGLE drafts at the feature level — one step ahead in the model’s latent space rather than the token space — and reliably achieves higher acceptance rates than token-level drafting. The simplest variant needs no training at all: n-gram or prompt-lookup speculation proposes tokens copied from the prompt itself or from recent output. For extractive workloads like summarization, code editing, or multi-turn conversations where much of the answer echoes the input, this “draft by copy-paste” approach is shockingly effective and costs nothing to set up.
Choosing a draft source is a workload decision:
- Extractive or templated output → n-gram/prompt-lookup. Zero training, near-free, high acceptance.
- General chat/agent workloads with one model family → a small sibling model or self-drafting heads.
- Maximum acceptance on a fixed model you control → EAGLE-style feature-level drafting, if you can run the fine-tuning.
Turning It On
Serving stacks expose speculation as configuration. In vLLM, you pass a speculative config at engine startup, selecting the method, the drafting model or head set, and how many tokens to propose per cycle. A typical setup looks like this:
from vllm import LLM
llm = LLM(
model="meta-llama/Llama-3.1-8B-Instruct",
speculative_config={
"method": "ngram",
"num_speculative_tokens": 5,
},
)The exact keys vary by engine version — check your engine documentation for the method names and options supported in your build. The more important engineering question is not how to enable it but how to measure it. Acceptance rate and time-per-token are the two numbers that tell you whether speculation is paying for itself. A minimal client-side harness makes the before/after honest:
package main
import (
"bufio"
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
)
func main() {
baseURL := os.Getenv("LLM_BASE_URL")
apiKey := os.Getenv("LLM_API_KEY")
body := map[string]any{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"stream": true,
"max_tokens": 512,
"messages": []map[string]string{
{"role": "user", "content": "Summarize this changelog entry in three bullets."},
},
}
payload, err := json.Marshal(body)
if err != nil {
fmt.Fprintln(os.Stderr, "marshal:", err)
os.Exit(1)
}
req, err := http.NewRequest(http.MethodPost, baseURL+"/chat/completions", bytes.NewReader(payload))
if err != nil {
fmt.Fprintln(os.Stderr, "request:", err)
os.Exit(1)
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+apiKey)
start := time.Now()
resp, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Fprintln(os.Stderr, "do:", err)
os.Exit(1)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
b, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
fmt.Fprintf(os.Stderr, "status %d: %s\n", resp.StatusCode, b)
os.Exit(1)
}
chunks := 0
scanner := bufio.NewScanner(resp.Body)
for scanner.Scan() {
if strings.HasPrefix(scanner.Text(), "data: ") {
chunks++
}
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "stream:", err)
os.Exit(1)
}
elapsed := time.Since(start)
perChunk := elapsed
if chunks > 0 {
perChunk = elapsed / time.Duration(chunks)
}
fmt.Printf("chunks=%d elapsed=%s per_chunk=%s\n", chunks, elapsed, perChunk)
}
Run it against the same engine with speculation off and on, same prompt set, same sampling parameters. Chunks map roughly to tokens, so per_chunk is your decode latency proxy. If the speedup is under ~1.2×, your workload isn’t accepting drafts — try a longer speculation window before giving up, and watch engine-side acceptance-rate metrics if your deployment exposes them.
When the Speedup Is Real — and When It Isn’t
Speculative decoding helps most under three conditions: low request concurrency (spare compute actually exists), predictable output (drafts get accepted), and latency-sensitive serving (you care about time-per-token, not raw hourly throughput). It hurts or does nothing when the batch is already large — verification tokens inflate every pass and steal bandwidth from other requests — and when output is highly creative, so acceptance collapses and you pay the draft cost for nothing.
Treat it as a per-deployment toggle, not a global default. The responsible rollout is: benchmark with your own prompts, enable for the latency-sensitive tier, and leave the throughput tier alone. Teams that skip the measurement step routinely “enable acceleration” and then wonder why p99 got worse during the morning peak.
Speculative decoding isn’t exotic anymore — it’s a config flag in every major serving stack. The interesting work has moved to the draft side: better self-drafting architectures, drafting with retrieval hints, and adaptive windows that size speculation to live acceptance rates. If you serve open-weight models, an afternoon of benchmarking will tell you whether you’ve been leaving a 2× on the table.