⌘K
ENES
GitHub
conceptSistemas de Inferencia~1 min de lecturaActualizado 2026-06-07#inference#speculative-decoding#decoding
Esta nota todavía no está traducida, así que se muestra la fuente en inglés.

Speculative decoding

Speculative decoding uses a cheaper draft model to propose several next tokens, then asks the larger target model to verify them. If the draft is often right, the system gets multiple accepted tokens per target-model step.

Basic flow

  1. The draft model generates a short candidate continuation.
  2. The target model evaluates those candidate tokens.
  3. Accepted tokens are emitted.
  4. Rejected tokens are corrected by the target model.
  5. The loop repeats until completion.

The output distribution can remain equivalent to the target model when implemented with the right acceptance rule.

When it helps

Helps when Hurts when
draft model is much faster draft is too inaccurate
target model is expensive verification overhead dominates
output is predictable task requires surprising or high-entropy tokens
serving stack supports it implementation increases complexity

Design choices

  • Draft model size and quality.
  • Number of speculative tokens per step.
  • Whether draft and target share tokenizer.
  • Hardware placement for draft and target.
  • Monitoring accepted-token rate by task.

Pitfall

Speculative decoding is not magic compression. If the draft model predicts poorly for your workload, you add complexity without reducing latency or cost.

Connects to: decoding and sampling · latency · serving engines

Notas relacionadas

Sistemas de InferenciaBatching for LLM servingBatching improves GPU utilization, but LLM serving needs dynamic and continuous batching because requests have different prompt and output lengths.Sistemas de InferenciaCost modeling for LLM servingLLM serving cost models connect traffic, token counts, model size, hardware, utilization, cache rate, retries, and quality gates.Sistemas de InferenciaFlashAttention and efficient attentionFlashAttention speeds transformer attention by reducing memory reads and writes, making attention more IO-aware.Sistemas de InferenciaGPU and hardware basicsLLM inference is often limited by VRAM capacity, memory bandwidth, interconnects, utilization, and the shape of prefill versus decode workloads.