What is Speculative Decoding and When Does It Help?

Speculative decoding accelerates autoregressive generation by using a fast draft model to propose multiple tokens, then verifying them in a single forward pass with the larger target model. When the draft is correct, multiple tokens are accepted at once, reducing the number of expensive target model iterations. Crucially, this is a lossless optimization that preserves the exact output distribution of the target model because incorrect draft tokens are rejected and regenerated. The approach works because the draft model generates k tokens per sequence, typically 4 to 8, much faster than the target model could. The target model then constructs a verification input that aligns these draft tokens so it can evaluate all of them in one forward pass. If the draft matches what the target model would have produced, all k tokens are accepted. If there is a mismatch at position i, tokens up to position i minus 1 are accepted and generation continues from the corrected token at position i. The speedup depends critically on the acceptance rate and verification efficiency. Production systems report 1.5 times to 2.5 times throughput gains for decode dominated workloads when acceptance rates are high. OpenAI's published work shows that with efficient verification kernels and a well matched draft model, speculative decoding delivers these speedups without changing output quality. The technique is most effective when decode latency dominates total latency, when the draft model fits in the memory budget alongside the target model, and when the scheduler can batch verification steps efficiently. The tradeoff is complexity versus speedup. Speculative decoding adds another model to deploy, extra control flow for verification, and additional KV cache management because both draft and target need cache. If the acceptance rate drops below 30 to 40 percent due to domain shift, strict syntax requirements in code generation, or long range reasoning tasks, the verification overhead can outweigh the benefits and speculative decoding becomes a net negative. Colocating draft and target on the same device avoids PCIe bottlenecks but requires careful memory planning. Reusing KV blocks across verification and subsequent decode is essential to avoid redundant computation.

Draft model proposes 4 tokens in 20ms, target verifies in 80ms (100ms total). Acceptance rate 60% yields 2.4 tokens per 100ms versus 2 tokens per 160ms (2 target iterations), giving 1.92x speedup

OpenAI research: Efficient verification with KV reuse shows lossless 1.5x to 2x gains on dialog tasks with high acceptance rates for well matched draft models

Pathological case: Code generation with strict syntax has 25% acceptance rate. 4 token draft takes 20ms, verification 80ms, but only 1 token accepted. 100ms per token versus 80ms without speculation is a net loss

Memory planning: 7B target model uses 14 GB, 1B draft model uses 2 GB, combined KV cache for both needs 40 GB, fits on single 80 GB GPU with colocation