Every other technique in this track makes a model cheaper by making it worse, if only slightly. Speculative decoding does not. It is a pure latency win, and the reason it works comes down to how transformers spend their time.
The inefficiency it exploits
Generation is sequential. Each token depends on the one before, so a large model produces output one forward pass at a time. Those passes are dominated by moving the model's weights through memory rather than by arithmetic.
The consequence is that a forward pass processing one token and a forward pass processing five tokens cost almost the same, because the expensive part is loading the weights either way. Sequential generation leaves nearly all of that capacity unused.
The mechanism
- A small, fast draft model generates several tokens ahead, cheaply
- The large model processes that entire proposed sequence in one forward pass
- Tokens matching what the large model would itself have produced are accepted
- At the first divergence, the large model's own token is used and the rest discarded
- Repeat
When the draft is right, you got several tokens for one expensive pass. When it is wrong, you got one, which is what you would have had anyway. The downside is bounded and the upside is not.
Why the output is unchanged
The verification step is the whole trick. Nothing is accepted that the large model would not have produced, so the final distribution is identical. You are not approximating the large model. You are running it, with the wasted capacity in each pass used to check guesses.
This makes speculative decoding uniquely safe to deploy. There is no quality evaluation to run afterwards, because there is no quality change to detect.
Choosing a draft model
The economics turn on acceptance rate. A draft model that is too weak gets rejected constantly and adds overhead for nothing. One that is too strong costs nearly as much as the model it is drafting for.
The sweet spot is a much smaller model that agrees with the large one most of the time, which is exactly what distillation produces. A student distilled from the target model makes an unusually good draft model, since it was explicitly trained to imitate the thing it is now guessing on behalf of. See What Is Knowledge Distillation?
Where it fits
It composes with everything else. Quantise the large model, quantise the draft model, and run speculative decoding between them. Because it changes no output, it is generally the first optimisation to reach for and the least risky to keep.
Common questions
What is speculative decoding?
Speculative decoding uses a small draft model to propose several tokens at once, then verifies them in a single forward pass of the large model. Accepted tokens are kept and the first rejection is corrected, so several tokens can be produced for roughly the cost of one large-model step.
Does speculative decoding change the output?
No. Correctly implemented, it produces exactly the distribution the large model would have produced alone. This is what distinguishes it from quantization, pruning and distillation, all of which trade some quality for speed.