⌘K
ENES
GitHub
conceptSistemas de Inferencia~1 min de lecturaActualizado 2026-06-07#inference#batching#serving#throughput
Esta nota todavía no está traducida, así que se muestra la fuente en inglés.

Batching for LLM serving

Batching lets the GPU process multiple requests together. For LLMs, batching is tricky because each request has different prompt length, output length, arrival time, and stopping condition.

Batching types

Type How it works Tradeoff
Static batching fixed batch formed before processing simple but wastes capacity
Dynamic batching briefly waits to group arrivals higher throughput, possible TTFT delay
Continuous batching adds and removes requests during generation high utilization, complex scheduler

Continuous batching is valuable because decode steps are sequential and requests finish at different times.

What batching optimizes

  • GPU utilization.
  • Tokens/sec across all users.
  • Cost per generated token.
  • Serving capacity under bursty traffic.

It can also increase queueing delay if the scheduler waits too long to form efficient batches.

Scheduler considerations

  • Separate prefill and decode workloads where useful.
  • Avoid one long request blocking many short requests.
  • Track per-tenant fairness and priority.
  • Combine batching with KV-cache management.
  • Monitor TTFT and p95, not only throughput.

Pitfall

Batching is not a free win for interactive products. A system can be efficient and feel slow if users spend too long waiting for their first token.

Connects to: latency vs throughput · KV cache · serving

Notas relacionadas

Sistemas de InferenciaLatency vs throughputLLM serving separates user-facing latency from system throughput, with different bottlenecks in prefill, decode, TTFT, and tokens per second.Sistemas de InferenciaServing enginesServing engines such as vLLM, TGI, and TensorRT-LLM package scheduling, batching, KV-cache management, kernels, APIs, and deployment concerns.Sistemas de InferenciaWhy inference is the real costTraining is a large upfront expense, but inference repeats for every user request and often dominates product unit economics.Sistemas de InferenciaCost modeling for LLM servingLLM serving cost models connect traffic, token counts, model size, hardware, utilization, cache rate, retries, and quality gates.