Applies toSelf-hosted inference with vLLM, SGLang or TensorRT-LLM; draft models, n-gram/prompt lookup, EAGLE/Medusa-style heads where supported
How it works
- A draft (small model, n-gram lookup or extra prediction heads) proposes the next k tokens.
- The target model checks all k in one forward pass.
- Accepted tokens are kept; generation continues from the first rejected one.
With the standard verification method, output follows the target model's distribution, so quality is unchanged.
Draft options
| Method | Needs | Works best for |
|---|---|---|
| Small draft model (same tokenizer/family) | Extra GPU memory for the draft | General text |
| N-gram / prompt lookup | Nothing extra | Outputs that copy from the input (RAG answers, code edits, summaries) |
| EAGLE / Medusa-style heads | Trained heads for the target model | High acceptance when available |
Enable (vLLM example)
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--speculative-config '{"model": "meta-llama/Llama-3.2-1B-Instruct", "num_speculative_tokens": 5}'
# n-gram prompt lookup, no draft model
vllm serve <model> --speculative-config '{"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_max": 4}'
Configuration syntax changes between vLLM versions; check the docs for yours.
Measure
| Metric | Why |
|---|---|
| Acceptance rate | Low acceptance means wasted draft work |
| Inter-token latency p50/p95 | Main benefit |
| Throughput at target concurrency | Can drop under heavy batching |
| GPU memory | Draft model uses KV cache space |
Test at low and high concurrency. Speculative decoding helps most when the GPU has spare compute (low to moderate batch sizes). At high batch sizes the GPU is already busy and the gain can vanish or turn negative.
Where it fits
| Situation | Worth trying |
|---|---|
| Interactive chat with latency targets, moderate load | Yes |
| RAG answers quoting retrieved text | Yes, n-gram lookup |
| Large offline batch jobs maximising throughput | Usually not |
| GPU memory already tight | Draft model may not fit; try n-gram |
Tip
Start with n-gram prompt lookup on RAG and summarisation workloads. It needs no extra model and often gives a quick latency win.
Checklist
- Latency goal and current baseline measured
- Draft method chosen for workload type
- Acceptance rate and latency measured at real concurrency
- Throughput and memory impact checked
- Kept only where it improves targets