Runbook: rolling out a new vLLM version or model on Kubernetes without dropping in-flight streams

#The problem

Every vLLM rollout or node drain produces a burst of user complaints: answers stop mid-sentence, clients get connection reset or 502, agents lose a tool-planning step. Nothing crashed. Kubernetes replaced pods that were still streaming tokens.

Three things make LLM pods harder to replace than normal web pods:

• Requests are long. A 4,000-token answer at 40 tokens/s is 100 seconds.
• Startup is slow. Loading weights and capturing CUDA graphs can take minutes, so a new pod is not ready when the old one dies.
• Capacity is scarce. There may be no spare GPU for a surge pod.

This runbook makes replacement safe without depending on how a particular vLLM version handles SIGTERM. Shutdown behaviour has changed across releases, so the drain below waits on vLLM's own metrics instead.

#Step 1: measure your longest request

Size the grace period from data, not guesses:

histogram_quantile(0.999, sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[24h])))

If p99.9 end-to-end latency is 140 s, the drain window needs to be about 180 s. Check the metric name against curl -s localhost:8000/metrics | grep e2e; names vary by version.

#Step 2: decide the rollout strategy from GPU headroom

Check headroom before starting:

kubectl get nodes -l nvidia.com/gpu.present=true \
  -o custom-columns=NODE:.metadata.name,ALLOC:.status.allocatable.nvidia\\.com/gpu
kubectl describe nodes -l nvidia.com/gpu.present=true | grep -A2 "Allocated resources" | grep nvidia

Warning

With maxSurge: 1 and no free GPU, the surge pod stays Pending and the rollout stalls. That is safe, but if someone "fixes" it by switching to maxUnavailable: 1 at peak, the remaining pods take the full load and queue times jump.

#Step 3: readiness, startup and liveness probes

startupProbe:
  httpGet: { path: /health, port: 8000 }
  periodSeconds: 10
  failureThreshold: 90        # up to 15 minutes to load weights
readinessProbe:
  httpGet: { path: /health, port: 8000 }
  periodSeconds: 5
  failureThreshold: 2
livenessProbe:
  httpGet: { path: /health, port: 8000 }
  periodSeconds: 20
  failureThreshold: 6

The startup probe stops liveness from killing a pod that is still loading. Readiness gates traffic, so a new pod receives nothing until the model is loaded.

For stricter readiness, add a warm-up call: a small postStart or sidecar that sends one completion request and creates a file the readiness probe checks. That prevents the first real user paying for CUDA graph capture or first-request compilation.

#Step 4: the preStop drain

When a pod starts terminating, Kubernetes removes it from Service endpoints and runs preStop before sending SIGTERM. Endpoint removal propagates to ingress controllers and kube-proxy asynchronously, so new requests can arrive for a few seconds. The drain below waits for that, then waits until vLLM reports no running or waiting requests.

spec:
  terminationGracePeriodSeconds: 200
  containers:
  - name: vllm
    lifecycle:
      preStop:
        exec:
          command:
          - python3
          - -c
          - |
            import time, urllib.request
            time.sleep(10)                      # let endpoint removal propagate
            deadline = time.time() + 170
            def active():
                txt = urllib.request.urlopen("http://localhost:8000/metrics", timeout=5).read().decode()
                n = 0.0
                for line in txt.splitlines():
                    if line.startswith(("vllm:num_requests_running", "vllm:num_requests_waiting")):
                        n += float(line.rsplit(" ", 1)[1])
                return n
            while time.time() < deadline:
                try:
                    if active() == 0:
                        break
                except Exception:
                    break
                time.sleep(2)

Rules for the numbers:

• terminationGracePeriodSeconds > sleep + drain deadline + a margin for vLLM's own shutdown. The grace period covers preStop and the time after SIGTERM.
• The drain deadline should be at or above your p99.9 request time from Step 1.
• If the pod runs several engines (data parallel) the metrics carry labels; the loop sums them all.

Caution

If preStop runs past the grace period, the kubelet sends SIGKILL and every remaining stream is cut. Do not set a drain deadline longer than the grace period allows.

#Step 5: check the path in front of the pods

The drain only helps if the proxies in front keep the existing stream open:

Also make sure the proxy does not retry a streaming request after it has started returning tokens. A retry there produces a duplicated or restarted answer.

#Step 6: protect against node drains

Rollouts are not the only replacement. Node upgrades and autoscaler scale-down evict pods too:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: llama-8b, namespace: llm }
spec:
  maxUnavailable: 1
  selector: { matchLabels: { app: llama-8b } }

kubectl drain respects PDBs and the pod's grace period. Check that your node upgrade tooling's drain timeout is longer than terminationGracePeriodSeconds, or it will force-delete pods.

#Step 7: run the rollout

kubectl -n llm set image deploy/llama-8b vllm=vllm/vllm-openai:<new-tag>
kubectl -n llm rollout status deploy/llama-8b --timeout=60m

# In a second terminal: watch drains and errors
kubectl -n llm get pods -l app=llama-8b -w
kubectl -n ingress-nginx logs deploy/ingress-nginx-controller -f | grep -E ' (499|502|504) '

While it runs, watch these in Prometheus:

sum(vllm:num_requests_waiting)
sum(rate(vllm:request_success_total[1m])) by (finished_reason)

A spike in waiting requests means you lost too much capacity at once. A change in finished reasons can show aborted requests; label names vary by version.

Roll back if p95 TTFT or error rate crosses your threshold:

kubectl -n llm rollout undo deploy/llama-8b

The rollback goes through the same drain, so it is also safe for streams.

#Step 8: make clients tolerant anyway

Some cut-offs will still happen (node failure, OOM). Clients should:

• Retry on connection errors only if no tokens were received yet.
• If tokens were received, show a clear "answer interrupted" state and offer to regenerate, rather than silently retrying.
• For agents, treat an interrupted step as failed and re-plan, with idempotency keys on any side-effecting tool.

#Prevent it next time

• Keep the grace period, drain deadline and proxy timeouts in one values file so they change together.
• Re-measure p99.9 request time after changing max_tokens limits or adding a reasoning model.
• Test the drain in staging: start a long stream, delete the pod, confirm the stream completes.
• Keep one GPU of headroom per model if rollouts during business hours are expected.