Speculative Decoding: Faster LLM Inference, Same Quality

If you have ever watched a large language model stream a response token by token, you have felt the bottleneck. The model is fast at math but slow at delivery, because it generates one token at a time and each token needs a full pass through the network. Speculative decoding is the trick that breaks this pattern. It can cut latency by two to three times on real workloads, and the output is provably identical to what the model would have produced on its own.

I spent a while assuming this was too good to be true. Faster generation with the same quality sounds like a free lunch, and free lunches usually hide a catch. The catch here is real but small, and once it clicks, the whole idea feels obvious. This post walks through what speculative decoding is, how it works, how much speedup you should actually expect, and where it quietly fails.

This is a companion to our main guide on LLM inference optimization, which covers the full landscape of techniques. Here we go deep on just one of them.

Why token-by-token generation is so slow

To understand what speculative decoding fixes, you have to understand the thing it works around: autoregressive generation.

A transformer generates text one token at a time. To produce token number 50, it needs tokens 1 through 49 as input. So generation is inherently sequential. You cannot compute token 50 and token 51 in parallel, because token 51 depends on knowing what token 50 turned out to be.

Here is the frustrating part. A single forward pass through a large model is not actually limited by compute. It is limited by memory bandwidth. Every token you generate requires reading the entire set of model weights from GPU memory. For a 70 billion parameter model, that is tens of gigabytes shuffled per token. The GPU's math units sit mostly idle while they wait for weights to arrive. You are paying the full cost of loading the model to produce a single token.

That imbalance is the whole opportunity. If a forward pass is bandwidth-bound and the compute is underused, then processing several tokens in one pass costs almost the same as processing one. The problem is that autoregressive generation gives you only one token to work on at a time. Speculative decoding changes that.

What is speculative decoding

A speculative decoding LLM setup uses two models instead of one. A small, fast draft model guesses several tokens ahead. Then the large target model checks all of those guesses in a single forward pass and keeps the ones it agrees with.

The mental model I find clearest is a fast typist working with a careful editor. The typist bangs out a whole sentence quickly, making a few likely mistakes. The editor reads the sentence in one glance, accepts the parts that are correct, fixes the first wrong word, and hands it back. The editor never has to type from scratch, so the pair moves much faster than the editor alone. But the final text is exactly what the editor would have written, because the editor signs off on every word.

That last point is what makes speculative decoding different from cheaper approximations like distillation or quantization. It is not trading quality for speed. The output distribution is mathematically identical to running the target model by itself. You are speeding up the same result, not settling for a worse one.

How does speculative decoding work

The loop has three parts: draft, verify, and correct.

The draft model proposes

The draft model is a much smaller version of the target, often 10 to 20 times smaller in model size. The draft model generates a short run of candidate tokens autoregressively, say 4 or 5 of them. Because it is small, these guesses are cheap. If your target is a 70B model, the draft might be a 7B or even a 1B model from the same family, so it shares vocabulary and tends to agree with the target on easy tokens.

The target model verifies in parallel

Now the target model takes the original prompt plus all the drafted tokens and runs a single forward pass over the whole sequence. This is the key move. Because the drafted tokens already exist, the target can compute its own probability for each position at the same time, in parallel, using the same batched matmul it would use for one token. One expensive pass gives you verification of several tokens.

The correction step keeps quality identical

For each drafted token, the algorithm compares the draft model's probability against the target model's probability and applies an accept or reject rule. Tokens the target agrees with are accepted. At the first token the target rejects, that token is resampled from a corrected distribution and the rest of the draft is thrown away. This acceptance rule, introduced in the speculative sampling work from Google DeepMind and in parallel by researchers at Google Brain, is designed so the accepted sequence follows exactly the target model's distribution.

So in the best case, all 5 drafted tokens get accepted and you produced 5 tokens for the price of roughly one target pass plus one cheap draft run. In the worst case, the very first token gets rejected and you fall back to normal speed, having wasted a little draft compute. On average, real workloads land somewhere comfortable in between.

A worked example with real numbers

Say the draft model proposes these 5 tokens after the prompt "The capital of France is":

Draft proposes:  " Paris" " ," " a" " city" " of"

The target model verifies all five in one pass and reports it agrees with the first four but would have chosen " known" instead of " of" at position five.

Target accepts:  " Paris" " ," " a" " city"   [reject at " of"]
Target resamples: " known"

You just emitted five correct tokens (" Paris", " ,", " a", " city", " known") using one target forward pass instead of five, roughly one cheap check per output token. The rejected " of" cost you nothing beyond a bit of draft work. That is a 5x reduction in expensive passes for this step. String enough of these together and the end-to-end latency drops sharply.

The number that governs everything is the acceptance rate: what fraction of drafted tokens the target keeps. High acceptance means big speedups. Low acceptance means you are drafting tokens that get thrown away, and the overhead can even make you slower than plain decoding.

The variants you will run into

"Speculative decoding" is really a family of techniques. The differences come down to where the draft comes from.

The classic approach uses a separate small draft model, as described above. It is simple and effective but requires you to have a smaller model in the same family, and you pay to load two models.

Self-speculative decoding drops the separate draft model. The target model drafts for itself by skipping some of its own layers, then verifies with the full stack. No second model to host.

Medusa attaches extra lightweight prediction heads to the target model so it can propose several future tokens at once, then verifies them together with a tree-based attention scheme.

EAGLE predicts at the feature level rather than the token level and has become one of the stronger methods in recent benchmarks, though as the Semrush data shows, "eagle speculative decoding" carries a high difficulty to rank for as a search term precisely because it is a hot research area.

Prompt lookup decoding, sometimes called n-gram speculation, skips models entirely. It drafts by copying likely continuations straight from the prompt. This works shockingly well for tasks like summarization or code editing where large chunks of the output are lifted from the input.

How does speculative decoding perform in practice

Reported speedups from the original speculative sampling paper landed around 2x to 2.5x on the models tested, and later methods like EAGLE have pushed that higher on some benchmarks. But the honest answer is that your mileage depends on three things.

First, the acceptance rate. The better your draft model predicts the target, the more tokens survive verification. A well-matched draft from the same model family accepts a large majority of tokens on ordinary text. A poorly matched draft accepts few and can slow you down.

Second, the task. Predictable text such as boilerplate code, structured formats, or continuations of the prompt accepts at very high rates. Creative or high-entropy text where many tokens are plausible accepts at lower rates, because the target is pickier.

Third, the draft length. Guessing more tokens per round raises the ceiling on speedup but also raises the cost of a wrong guess, since everything after the first rejection is discarded. Most systems tune this to a small number like 3 to 7. Some modern serving stacks adjust it dynamically based on recent acceptance.

Speculative decoding in vLLM and other serving stacks

You rarely implement the accept-reject math yourself. Modern inference servers bake it in. vLLM speculative decoding, for example, is a configuration option rather than a research project. You point the server at a target model and a draft model and it handles the loop.

from vllm import LLM, SamplingParams

# Target model does the verifying, draft model does the guessing.
llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_config={
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 5,
    },
)

params = SamplingParams(temperature=0.7, max_tokens=256)
output = llm.generate("Explain speculative decoding in one paragraph.", params)
print(output[0].outputs[0].text)

The important thing to notice is that nothing about your sampling parameters changes. Temperature, top-p, and max tokens all behave exactly as they would without speculation, because the output distribution is preserved. Speculative decoding is a serving-layer optimization that is invisible to the caller. That is also why it composes cleanly with other techniques in the AI cost optimization toolkit, and why it sits alongside batching and cache management in a well-tuned model serving setup.

When speculative decoding is not worth it

The technique has clear failure modes, and knowing them saves you from a disappointing benchmark.

It hurts when the acceptance rate is low. If your draft model rarely agrees with the target, you are spending draft compute and verification overhead to accept almost nothing. High-temperature sampling on creative tasks can push acceptance down far enough that plain decoding wins.

It hurts at very high batch sizes. Speculative decoding trades spare compute for fewer sequential steps. When you are serving many requests at once, the GPU is already busy and there is little idle compute to exploit. Under heavy concurrency, the extra draft work can cost more than it saves. This interacts directly with continuous batching, which is why serving stacks sometimes disable speculation once batch size crosses a threshold.

It adds memory pressure. Hosting a second model, or extra prediction heads, uses GPU memory you could have spent on a larger KV cache or more concurrent requests. On a tight memory budget, that trade may not be worth it.

Common mistakes I made

The first time I set this up, I grabbed a draft model from a different model family than the target. The vocabularies did not line up cleanly and the acceptance rate was terrible. Use a draft from the same family as the target so they share a tokenizer and tend to agree.

I also tuned the number of speculative tokens too high, thinking more guesses meant more speed. On my workload, a longer draft meant more tokens discarded after the first rejection, and throughput dropped. Start small, measure acceptance, and only grow the draft length if acceptance stays high.

Finally, I benchmarked at batch size one, got excited about a 2.4x speedup, then watched it evaporate in production. Speculative decoding shines for latency-sensitive, low-concurrency serving. If your goal is maximum throughput under heavy load, look harder at batching before you reach for speculation.

Wrapping up

Speculative decoding earns its reputation. It is one of the rare optimizations that is genuinely free on the quality axis, because the math guarantees the same output. The real work is in matching a good draft model to your target, tuning the draft length, and measuring acceptance on your own traffic so you know whether the speedup survives your production batch sizes.

If you want the bigger picture of where this fits, our LLM inference optimization guide lays out the full set of levers, from prompt caching to quantization to batching, and how they stack together. Speculative decoding is one of the sharpest tools in that box.

For the foundational algorithm, the original papers are worth reading: "Fast Inference from Transformers via Speculative Decoding" by Leviathan, Kalman, and Matias, and "Accelerating Large Language Model Decoding with Speculative Sampling" from Google DeepMind. The vLLM documentation covers how to turn it on in production.

This post is part of the Levelop series on making AI systems faster and cheaper to run.