Where GPU memory goes
At start-up vLLM claims --gpu-memory-utilization of the GPU (0.9 by default). It loads the model weights, keeps some space for activations, and gives the rest to the KV cache.
Weights ≈ parameters × bytes per parameter:
| Model size | FP16/BF16 | FP8 | INT4 (AWQ/GPTQ) |
|---|---|---|---|
| 8B | ~16 GB | ~8 GB | ~5 GB |
| 70B | ~140 GB | ~70 GB | ~40 GB |
KV cache per token = 2 × layers × KV heads × head dim × bytes.
For Llama 3.1 8B (32 layers, 8 KV heads, head dim 128, BF16): 2 × 32 × 8 × 128 × 2 = 128 KiB per token. One 8k-token conversation uses about 1 GiB of KV cache. That's why concurrency is limited by memory, not compute.
Start-up flags that matter
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--gpu-memory-utilization 0.90 \
--max-model-len 16384 \
--max-num-seqs 128 \
--tensor-parallel-size 1 \
--enable-prefix-caching \
--kv-cache-dtype fp8
| Flag | Effect |
|---|---|
--max-model-len | Maximum context. Lower it if start-up fails with "not enough KV cache". |
--max-num-seqs | Maximum concurrent sequences per batch. Higher gives more throughput but worse per-request latency. |
--tensor-parallel-size | Split the model across N GPUs in one node. Needed when weights don't fit on one GPU. |
--enable-prefix-caching | Reuses KV cache for shared prompt prefixes such as system prompts and RAG templates. Often a big TTFT improvement. |
--kv-cache-dtype fp8 | Roughly doubles KV cache capacity, with a small quality cost. Test it on your evals first. |
--quantization fp8 | FP8 weights on Hopper/Blackwell GPUs. About half the weight memory. |
Metrics to scrape
vLLM exposes Prometheus metrics at /metrics. Exact names change between versions, so check your build's output. The important ones:
| Metric | Watch for |
|---|---|
vllm:num_requests_running | Close to max-num-seqs → batch is full |
vllm:num_requests_waiting | Above 0 for long periods → you need more replicas |
KV cache usage (vllm:gpu_cache_usage_perc / vllm:kv_cache_usage_perc) | Near 100% → preemptions and latency spikes |
vllm:num_preemptions_total | Increasing → sequences evicted and recomputed. Lower concurrency or add memory. |
vllm:time_to_first_token_seconds | p95 is what users feel first |
vllm:time_per_output_token_seconds / inter-token latency | Streaming smoothness |
Also watch GPU health:
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw --format=csv -l 5
Scaling rules of thumb
- Scale out on queue depth (
num_requests_waiting), not on GPU utilization. GPU utilization looks high even when the server is coping fine. - A KV cache that's frequently full is a capacity signal. Add replicas, use FP8 KV cache, or lower
max-model-lenif most requests are short. - Keep at least two replicas behind the load balancer, so a model reload or GPU XID error doesn't cause an outage.
Common failures
| Symptom | Likely cause |
|---|---|
| CUDA OOM at start-up | Weights + KV cache don't fit. Lower max-model-len, raise TP, or quantize. |
| Start-up takes many minutes | Downloading weights every time. Cache them on a persistent volume or local NVMe. |
| Latency spikes under load | Preemptions from a full KV cache, or long prompts blocking the queue. Turn on chunked prefill (default in recent versions). |
NCCL errors with TP > 1 | GPUs not in the same NVLink domain, or a shared-memory limit in containers (--shm-size) |
Tip
Before you go live, load-test with your real prompt-length distribution. A 200-token chat benchmark tells you little about 6,000-token RAG prompts.