Applies tovLLM 0.6+ (metric names vary by version), Text Generation Inference and similar servers on GPU; Kubernetes or VMs
What TTFT is made of
TTFT = time waiting in queue + prefill time (processing the prompt) + first decode step
A spike is almost always one of: more requests than capacity, much longer prompts, or KV cache full (requests preempted or waiting for memory).
Step 1: look at the server metrics
vLLM exposes Prometheus metrics at /metrics:
curl -s http://vllm:8000/metrics | grep -E "num_requests_(running|waiting)|cache_usage|time_to_first_token|prompt_tokens|preempt"
| Metric (names vary by version) | Tells you |
|---|---|
vllm:num_requests_waiting | Queue length. Growing means not enough capacity |
vllm:num_requests_running | Concurrent sequences in the batch |
vllm:gpu_cache_usage_perc / vllm:kv_cache_usage_perc | KV cache fill. Near 100% means memory pressure |
vllm:num_preemptions_total | Requests evicted from KV cache and recomputed |
vllm:time_to_first_token_seconds (histogram) | The symptom, by percentile |
vllm:request_prompt_tokens (histogram) | Prompt length distribution |
Step 2: match the pattern
| Pattern | Cause | Fix |
|---|---|---|
| Waiting queue grows, cache < 90% | Not enough replicas for the request rate | Scale out; check autoscaler triggers on queue length, not CPU |
| Cache ≈ 100%, preemptions rising | KV cache too small for concurrent long sequences | Lower --max-num-seqs, limit max_tokens, use a larger GPU / tensor parallelism, FP8 KV cache |
| Prompt token p95 jumped | A client started sending huge prompts (whole documents, long chat history) | Cap input length at the gateway; fix the client (summarise history, fewer RAG chunks) |
| Spikes when a long prompt arrives, others stall | Long prefills block decodes | Enable chunked prefill (default in newer vLLM) and tune --max-num-batched-tokens |
| Same system prompt / documents repeated | Prefill recomputed every time | Enable prefix caching (--enable-prefix-caching, default on in newer versions) |
| Only after deploy/restart | Cold start: CUDA graph capture, model load | Readiness probes that wait for warm-up; pre-warm requests |
Step 3: quick mitigations
- Shed load fairly: gateway rate limits per client; reject very long prompts with a clear error.
- Scale out if GPUs are available (see GPU capacity runbook if they're not).
- Route low-priority or batch traffic to another pool or a smaller model.
- Reduce
max_tokensfor chat UIs: long answers hold KV cache longer.
Step 4: the find-the-client query
If you log prompt token counts per caller at the gateway:
ApiManagementGatewayLlmLog
| where TimeGenerated > ago(1h)
| summarize p95_prompt = percentile(PromptTokens, 95), reqs = count() by CallerId = tostring(ApimSubscriptionId)
| order by p95_prompt desc
(Table and column names depend on how you log. The idea is prompt size by caller.)
Step 5: confirm recovery
TTFT p95 back under target, waiting queue near zero, preemptions flat. Keep the dashboard panels you used. They're the first thing to check next time.
Tip
Autoscale inference on queue length or KV cache usage, not GPU utilisation. A GPU can show 100% utilisation while serving well, or 60% while requests queue.
Prevent it next time
- SLOs for TTFT and inter-token latency, with alerts on p95.
- Input token caps per route at the gateway.
- Prefix caching and chunked prefill enabled and tested on your vLLM version.
- Load tests with realistic prompt length distributions before releases.