Most vLLM capacity problems are KV cache problems. The model fits, the first requests are fast, then under load latency jumps or requests are preempted. Knowing how vLLM budgets memory lets you tune it deliberately.
How vLLM uses GPU memory
At startup vLLM:
- Loads the weights.
- Runs a profiling pass with the maximum batch to measure activation memory.
- Gives everything left, up to
--gpu-memory-utilization(default 0.9) of total VRAM, to the KV cache, split into fixed-size blocks.
The startup log tells you the result. Look for lines like:
Model loading took 15.0 GiB
# GPU blocks: 9832, # CPU blocks: 2048
Maximum concurrency for 8192 tokens per request: 19.2x
(Wording varies by version.) That last number is what matters: with this config, about 19 requests at full 8k context fit at once.
The knobs
| Flag | Effect |
|---|---|
--gpu-memory-utilization | Fraction of VRAM vLLM may use; more = more KV cache |
--max-model-len | Longest prompt + output allowed per request |
--max-num-seqs | Max concurrent sequences scheduled |
--max-num-batched-tokens | Tokens processed per scheduler step (prefill chunk size) |
--kv-cache-dtype fp8 | Store KV cache in FP8, roughly half the memory |
--enable-prefix-caching | Reuse KV blocks for shared prompt prefixes (default on in recent versions) |
--enable-chunked-prefill | Split long prompts so decoding is not starved (default on in recent versions) |
--tensor-parallel-size | Shard the model over GPUs; frees memory per GPU for KV cache |
Size max-model-len from real traffic
The model may support 128k tokens, but if 99% of requests use under 6k, setting --max-model-len 131072 wastes nothing on its own (vLLM allocates blocks on demand), but it lets a single huge request take a big share of the cache and can force vLLM to refuse startup if even one max-length sequence cannot fit.
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 16384 \
--max-num-seqs 64 \
--gpu-memory-utilization 0.92 \
--kv-cache-dtype fp8 \
--enable-prefix-caching
Measure the token distribution from your gateway logs (prompt + completion tokens per request), then set max-model-len around the p99.5 with headroom, and route rare long-context requests to a separate deployment.
Estimate capacity yourself
kv_bytes_per_token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_value
tokens_in_cache = kv_cache_bytes / kv_bytes_per_token
concurrent_full = tokens_in_cache / max_model_len
Example: an 8B GQA model (32 layers, 8 KV heads, head_dim 128) has ~128 KiB/token in FP16, ~64 KiB in FP8. With 40 GiB for KV cache:
FP16: 40 GiB / 128 KiB ≈ 327k tokens -> ~20 sequences at 16k
FP8 : 40 GiB / 64 KiB ≈ 655k tokens -> ~40 sequences at 16k
Average requests are much shorter than the max, so real concurrency is higher.
Warning
FP8 KV cache saves memory with usually small quality impact, but not zero. Run your evaluation set before enabling it in production, and check that your GPU and model support it.
Preemption: the early warning
When the cache runs out mid-generation, vLLM preempts sequences (recomputes or swaps them later). Users see sudden latency spikes or stalls.
Watch the metrics endpoint (/metrics, Prometheus format):
curl -s localhost:8000/metrics | grep -E "num_requests_running|num_requests_waiting|gpu_cache_usage_perc|kv_cache_usage|num_preemptions"
| Metric (names vary by version) | Healthy | Trouble |
|---|---|---|
| KV cache usage % | Under ~90% at peak | Pinned near 100% |
| Requests waiting | Brief spikes | Steady queue |
| Preemptions total | Flat | Increasing |
| Time to first token p95 | Within SLO | Rising with queue |
If preemptions climb: lower --max-num-seqs, lower --max-model-len, enable FP8 KV cache, add tensor parallelism, or add replicas.
Prefix caching pays for shared prompts
RAG and agent apps resend the same long system prompt and tool definitions on every call. With prefix caching, those KV blocks are reused, cutting prefill time and memory. Make the shared part of the prompt identical and first (variable content later) to get hits. Hit rates appear in metrics (prefix_cache_hits/queries or similar).
Tensor parallel vs replicas
| Tensor parallel (TP=2 on 2 GPUs) | Two replicas (TP=1 each) | |
|---|---|---|
| Memory for KV cache | More per model instance | Less per instance |
| Single-request latency | Lower (work split) | Same as one GPU |
| Throughput | Good | Often better for small models |
| Failure blast radius | Both GPUs serve one instance | One replica can fail independently |
For models that fit comfortably on one GPU, replicas behind a load balancer usually give better throughput and resilience. Use TP when the model or the needed context does not fit.
A tuning loop
- Benchmark with realistic prompts and concurrency (
vllm bench serveor the benchmark scripts shipped with your version, or a load tool replaying real traffic). - Record TTFT p50/p95, inter-token latency, throughput (tokens/s), KV usage, preemptions.
- Change one flag, rerun.
- Keep settings in version-controlled deployment config with the benchmark results that justified them.
Prevent it next time
- Alerts on KV cache usage above 90% for 5 minutes and on any increase in preemptions.
- Separate deployments for long-context workloads.
- Re-run the benchmark after every vLLM upgrade; defaults change between releases.