On Speculative Decoding

If you’re here from Agentic Builders’ Collective, welcome! Add me on LinkedIn.

The slides are available under CC BY-SA 4.0 license.

Every figure in the deck is drawn from 199 benchmark runs on a single NVIDIA DGX Spark, provided by Ray Aun Fan and run autonomously by Claude Opus 4.8 and 5. The full results are public, and each number on a slide links back to the config page that produced it. The companion write-up is On Speculative Decoders.

The five rules

A speculative decoder is a small, cheap model that guesses the next few tokens so the big model can check several at once instead of producing them one at a time. The output is provably identical, and the speedups can be substantial. The catch is that it only pays off sometimes, and the conditions are narrower than the papers suggest. These five rules will help guide you on using this:

1. Drafters trade compute for speed. Gains are largest at low concurrency.
2. Drafters are brittle. The drafter is trying to match the target’s distribution, not just its tokens. Change the engine, the quant, the workload, or the draft weights, and the win can vanish. Test in the exact configuration you’ll run.
3. Agreement is critical. Acceptance rate affects how many tokens each decode pass gains.
4. Slower target, bigger relative win. The more expensive each target pass is, the more there is to amortize.
5. Speculation can’t fix broken. It is a multiplier, not a rescue. Get the quant and the engine right first, then add the drafter.

The short version: use the model’s own MTP path when it ships with one, and benchmark your own workload at the concurrency you actually run.

If you liked my talk, feel free to check out other talks I’ve given. If infrastructure is more your thing, Infrastructure-based Safety for Your ‘Claw was my last outing at ABC.