Speculative Decoding
Generating text is one expensive forward pass PER TOKEN — even for words a child could guess, like the ‘of’ in ‘capital of France’. Speculative decoding pairs a tiny fast DRAFTER with the big model as VERIFIER: the drafter guesses several tokens ahead, and the big model checks the whole chunk in a single pass.
Run normal decoding first and count the big-model passes. Then switch to speculative: the rabbit drafts four tokens, the elephant verifies them all at once, keeping the correct prefix and fixing the first wrong token. Easy stretches of text fly by; hard tokens fall back to the big model. The clever part: the accept/reject rule guarantees output identical to the big model alone — pure speed, no quality loss.
The tiny drafter guesses 4 tokens; the big model checks them all in ONE forward pass, keeping the correct prefix and fixing the first mistake. Easy words come cheap; the output is provably identical to normal decoding.
Check yourself
Why does verifying four drafted tokens cost the big model about the same as generating one?
Go deeper (free): Hugging Face — Speculative decoding (assisted generation) ↗