Monitor vLLM in production: Prometheus metrics, PromQL and alerts for queue, TTFT and KV cache

#When to use this

You run vLLM in production. GPU utilization looks high, and that tells you very little: a GPU can sit at 100% while users are either served well or waiting in a long queue. vLLM's own metrics show what users experience. This runbook covers what to scrape, which queries to use and how to read them.

#Step 1: check the metrics endpoint

vLLM's server exposes Prometheus metrics at /metrics by default:

curl -s http://vllm:8000/metrics | grep -E '^vllm:' | cut -d'{' -f1 | sort -u

Warning

Metric names change between vLLM releases. For example, GPU KV cache usage has appeared as both vllm:gpu_cache_usage_perc and vllm:kv_cache_usage_perc. Run the command above, and use the names that your version shows. Pin the version, and recheck dashboards after an upgrade.

Key metrics:

#Step 2: scrape it

Plain Prometheus:

scrape_configs:
  - job_name: vllm
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets: ["vllm-0:8000", "vllm-1:8000"]

With the Prometheus Operator on Kubernetes:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vllm
  namespace: inference
spec:
  selector:
    matchLabels:
      app: vllm
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

The Service's port must be named http for this to match. If vLLM requires an API key, check whether /metrics is exposed without auth, and restrict it at the network level. Also scrape GPU hardware metrics with DCGM exporter.

#Step 3: core PromQL

Queue and load:

# Waiting and running, by instance
sum by (instance) (vllm:num_requests_waiting)
sum by (instance) (vllm:num_requests_running)

Latency:

# P95 time to first token
histogram_quantile(0.95,
  sum by (le, model_name) (rate(vllm:time_to_first_token_seconds_bucket[5m])))

# P95 queue time
histogram_quantile(0.95,
  sum by (le) (rate(vllm:request_queue_time_seconds_bucket[5m])))

# P95 end-to-end latency
histogram_quantile(0.95,
  sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[5m])))

Throughput:

# Output tokens per second
sum(rate(vllm:generation_tokens_total[5m]))

# Input tokens per second
sum(rate(vllm:prompt_tokens_total[5m]))

# Completed requests per second, by finish reason
sum by (finished_reason) (rate(vllm:request_success_total[5m]))

KV cache:

max by (instance) (vllm:kv_cache_usage_perc)

Output truncation, as a share of finished requests:

sum(rate(vllm:request_success_total{finished_reason="length"}[15m]))
/ sum(rate(vllm:request_success_total[15m]))

A rising length share means answers are being cut off. See output truncated.

Label names can also vary by version. Check them in the raw /metrics output.

#Step 4: alert rules

Use these as a starting point, and set the thresholds from your own baseline and SLOs.

groups:
  - name: vllm
    rules:
      - alert: VllmDown
        expr: up{job="vllm"} == 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "vLLM {{ $labels.instance }} is not responding to scrapes"

      - alert: VllmQueueBuilding
        expr: sum by (instance) (vllm:num_requests_waiting) > 10
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "Requests are queuing on {{ $labels.instance }}"

      - alert: VllmTtftHigh
        expr: |
          histogram_quantile(0.95,
            sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 3
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "P95 TTFT is above 3 seconds"

      - alert: VllmKvCacheNearFull
        expr: max by (instance) (vllm:kv_cache_usage_perc) > 0.9
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "KV cache above 90% on {{ $labels.instance }}"

      - alert: VllmNoThroughputWithQueue
        expr: |
          sum(vllm:num_requests_waiting) > 0
          and sum(rate(vllm:generation_tokens_total[5m])) == 0
        for: 3m
        labels: { severity: critical }
        annotations:
          summary: "Requests are waiting, and no tokens are being generated"

The last alert catches a hung engine that still answers health checks. It's one of the most useful alerts here, because up stays at 1.

Tip

Keep the alert count low. Alert on what users feel, such as queue time and TTFT, and leave GPU temperature and utilization to dashboards unless hardware limits are near.

#Step 5: read the signals together

Separate queue time from processing time in any TTFT investigation:

histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
-
histogram_quantile(0.95, sum by (le) (rate(vllm:request_queue_time_seconds_bucket[5m])))

Subtracting percentiles gives only a rough figure, but it's enough to show direction. If most TTFT is queue time, you need capacity. If queue time is small, look at prompt length, prefix caching and GPU speed. See TTFT latency spikes.

Check average prompt and output size, because a traffic-shape change often looks like a capacity problem:

sum(rate(vllm:prompt_tokens_total[15m])) / sum(rate(vllm:request_success_total[15m]))
sum(rate(vllm:generation_tokens_total[15m])) / sum(rate(vllm:request_success_total[15m]))

#Step 6: incident steps

1. Open the dashboard, and find the matching row in the table above.
2. Check the changes: new model version, prompt template, config or traffic source.
3. Compare instances. If one replica is bad, remove it from rotation, and don't scale everything.
4. Collect the evidence before restarting:

kubectl logs -n inference deploy/vllm --since=15m | tail -200
kubectl exec -n inference deploy/vllm -- nvidia-smi
curl -s http://vllm:8000/metrics > vllm-metrics-$(date +%s).txt

5. Mitigate: limit traffic at the gateway, reduce max_tokens, or fail over to a backup.

Caution

Restarting a replica drops all its in-flight requests and its cache, and reloading large model weights can take minutes. Restart one at a time, and only after confirming there's enough capacity elsewhere. See the crash loop runbook.

#Scale on the right signal

GPU utilization makes a poor autoscaling metric, because it's already high at moderate load. Scale on vllm:num_requests_waiting, or on queue time. See KEDA and queue depth.

#Prevent it next time

• Load-test each model to find its max sustainable tokens per second, and set thresholds from it. See load testing.
• Pin vLLM's version, and keep dashboards in Git.
• Add a synthetic request every minute, which catches failures that metrics can't.
• Related: throughput tuning and end-to-end latency.