Home / AI & GenAI

AI & GenAIHow-toAdvanced

Optimization: vLLM throughput tuning - batching limits, prefix caching, chunked prefill and parallelism

A self-hosted model's cost per token depends on how many requests each GPU serves at once without breaking latency targets. Tune GPU memory use and batch limits, enable prefix caching and chunked prefill, choose tensor parallelism only when needed, and load-test against time-to-first-token and inter-token latency SLOs.

Deepak Bhardwaj3 min read

Applies tovLLM (recent releases) on NVIDIA GPUs, standalone or on Kubernetes; concepts also apply to SGLang and TensorRT-LLM

Set targets first

Metric Example target
Time to first token (TTFT) p95 < 1 s
Inter-token latency (ITL) p95 < 50 ms
Throughput Max tokens/sec per GPU while meeting the above

Tuning trades throughput against latency, so the targets decide the settings.

Key settings

Setting Effect Guidance
--gpu-memory-utilization Share of GPU memory vLLM uses (weights + KV cache) 0.85–0.95; leave headroom for other processes
--max-model-len Max context length Set to what you need; large values reduce concurrent sequences
--max-num-seqs Max concurrent sequences in a batch Raise for throughput until latency targets break
--max-num-batched-tokens Tokens processed per scheduler step Higher → more throughput, longer prefill stalls
--enable-prefix-caching Reuse KV cache for shared prompt prefixes On for chat/RAG with common system prompts
--enable-chunked-prefill Split long prompts so decoding continues Smoother ITL with long prompts (default on in newer versions)
--tensor-parallel-size Split model across GPUs Only when the model or KV cache doesn't fit one GPU
--kv-cache-dtype fp8 Smaller KV cache More concurrent sequences; check quality 
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.92 \
  --max-model-len 16384 \
  --max-num-seqs 128 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8

Flag names and defaults change between versions; check vllm serve --help for yours.

Load-test properly

  • Use realistic prompt and output length distributions, not fixed short prompts.
  • Ramp concurrency and record TTFT, ITL and tokens/sec at each level.
  • vLLM's benchmark scripts or tools like GuideLLM can drive these tests.

Watch the metrics

From the /metrics endpoint (see the vLLM metrics post):

Metric Signal
KV cache usage Near 100% → requests queue or get preempted
Requests waiting Growing queue → add replicas or reduce max-model-len
Preemptions KV cache pressure; reduce concurrency or context length
Prefix cache hit rate Is prefix caching helping?

Scale decisions

Situation Action
One GPU underused at target latency Raise max-num-seqs
KV cache full, latency rising Quantized KV cache, shorter max-model-len, or more replicas
Model doesn't fit one GPU Tensor parallel within a node, or a quantized model (see the FP8/INT4 transformation post)
Steady growth More replicas behind a load balancer before larger TP sizes

Tip

Several replicas with tensor-parallel 1 often give more total throughput than one replica spread across several GPUs, if the model fits in one GPU's memory.

Checklist

  • Latency targets defined
  • Memory, context and batch settings tuned
  • Prefix caching and chunked prefill enabled
  • Tensor parallelism only when needed
  • Realistic load tests at rising concurrency
  • KV cache, queue and preemption metrics monitored

Related fixes