Sev1: vLLM pods crash-loop with CUDA out of memory and the self-hosted LLM endpoint is down

#Symptoms

• Pods in CrashLoopBackOff; logs end with one of:
  • torch.OutOfMemoryError: CUDA out of memory. Tried to allocate ...
  • ValueError: No available memory for the cache blocks. Try increasing gpu_memory_utilization when initializing the engine.
  • ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (...)
• Or the server starts but dies with OOM under load.
• The gateway returns 502/503 for every model call.

#Step 1: restore service (10 minutes)

Roll back to the last known-good deployment (image tag, model revision, flags):

kubectl -n llm rollout history deploy/llama-8b
kubectl -n llm rollout undo deploy/llama-8b
kubectl -n llm rollout status deploy/llama-8b --timeout=15m

If nothing was deployed (OOM under load only), reduce load on the engine while you investigate:

• Route part of the traffic to a fallback model or hosted API at the gateway.
• Lower concurrency for the next start: --max-num-seqs 32 and a lower --max-model-len.

#Step 2: read the startup log

kubectl -n llm logs deploy/llama-8b --previous | grep -Ei "model loading took|gpu blocks|kv cache|maximum concurrency|out of memory|max seq len" | tail -20

#Step 3: what changed?

Compare the failing and good specs:

kubectl -n llm rollout history deploy/llama-8b --revision=14 > good.txt
kubectl -n llm rollout history deploy/llama-8b --revision=15 > bad.txt
diff good.txt bad.txt

Usual culprits:

Check the GPU the pod got and what is on it:

kubectl -n llm get pod -l app=llama-8b -o wide
kubectl debug node/<node> -it --image=nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 -- nvidia-smi

#Step 4: fix the memory budget

Pick the smallest change that restores headroom:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 16384 \          # instead of the model's 128k default
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 64 \
  --kv-cache-dtype fp8 \           # if supported and evaluated
  --tensor-parallel-size 2         # if the model barely fits one GPU

Pin the GPU type so scheduling cannot pick a smaller card:

nodeSelector:
  nvidia.com/gpu.product: NVIDIA-L40S       # label from GPU feature discovery; values vary
resources:
  limits: { nvidia.com/gpu: 1 }

Warning

Do not "fix" OOM by setting --enforce-eager everywhere without measuring. It disables CUDA graphs, saving some memory but costing throughput and latency. Use it as a temporary measure only.

#Step 5: OOM under load (server was running)

If the engine started fine and crashed later, something outside the KV cache grew: very long prompts hitting a code path with big activations, multimodal inputs, or many LoRA adapters swapping. Check:

kubectl -n llm logs deploy/llama-8b --previous | grep -B5 -i "out of memory" | head -40

• Enforce a maximum prompt length at the gateway, below max-model-len.
• Lower --max-num-batched-tokens to cap prefill chunk size.
• Limit image count/size per request for multimodal models.

#Step 6: confirm

• Pods Ready and stable for 30 minutes under normal load.
• Startup log shows a reasonable KV capacity ("maximum concurrency" well above your peak concurrent requests).
• KV cache usage and preemption metrics normal; error rate at the gateway back to baseline.

#Prevent it next time

• Treat model, vLLM version and flags as one versioned unit; test new combinations in staging on the same GPU type.
• Explicit --max-model-len always; never rely on the model default.
• Alert on pod restarts with OOM and on KV cache usage above 90%.
• Gateway fallback route to another model or provider, tested regularly.