Applies toKubernetes GPU node pools on AKS (Spot VMs), EKS (Spot with Karpenter or managed node groups), GKE/OKE preemptible; self-hosted model servers (vLLM, TGI, Triton)
Symptoms
- Many GPU nodes disappeared within minutes; model pods
Pendingwith0/N nodes are available: insufficient nvidia.com/gpu. - The autoscaler logs capacity errors:
SkuNotAvailable,AllocationFailed,InsufficientInstanceCapacity,ZONE_RESOURCE_POOL_EXHAUSTEDor Karpenter "no instance type satisfied". - The inference endpoint returns 503; users of AI features see errors.
Step 1: fail traffic over (5 minutes)
Before fighting for capacity, keep users served:
- Switch the gateway route to a fallback: a hosted API (Azure OpenAI, Bedrock), a smaller model on remaining nodes, or another region.
- Turn on degraded mode in the app (shorter answers, fewer AI features) if you have a feature flag.
kubectl -n llm get pods -o wide | grep -v Running
kubectl get nodes -l workload=gpu-inference
Step 2: confirm it was spot eviction
kubectl get events -A --sort-by=.lastTimestamp | grep -Ei "preempt|evict|spot|interruption|NodeNotReady" | tail -30
- AKS: Activity Log entries for VMSS instance eviction; node pool
scaleSetPriority: Spot. - EKS: EC2 Spot interruption notices (EventBridge) and Karpenter
Interruptionevents.
az aks nodepool show -g rg-aks --cluster-name aks-ai -n gpuspot --query "{priority:scaleSetPriority, evictionPolicy:scaleSetEvictionPolicy, count:count, vm:vmSize}" -o table
kubectl -n karpenter logs deploy/karpenter | grep -i -E "interrupt|insufficient" | tail -20
Step 3: get GPU capacity back
Work through options quickly, most likely to succeed first:
| Option | How |
|---|---|
| On-demand in the same pool family | AKS: scale an existing on-demand GPU pool; EKS: allow on-demand capacity type in the NodePool |
| Other VM/instance sizes with the same or larger GPU | Add sizes to the allowed list |
| Other zones | Spread the pool or add a pool in another zone |
| Other region | Deploy the model where capacity exists; route traffic there |
| Smaller model / quantized variant | Fits on GPUs that are available |
AKS, on-demand pool:
az aks nodepool add -g rg-aks --cluster-name aks-ai -n gpuod \
--node-vm-size Standard_NC24ads_A100_v4 --node-count 2 \
--node-taints nvidia.com/gpu=present:NoSchedule --labels workload=gpu-inference \
--zones 1 2 3
# or scale an existing one
az aks nodepool scale -g rg-aks --cluster-name aks-ai -n gpuod --node-count 4
EKS with Karpenter, allow on-demand as a fallback:
requirements:
- { key: karpenter.sh/capacity-type, operator: In, values: ["spot", "on-demand"] }
- { key: karpenter.k8s.aws/instance-family, operator: In, values: ["g6e", "g6", "g5", "p4d"] }
Karpenter prefers spot when both are allowed and falls back to on-demand when spot is unavailable.
Warning
On-demand GPU quota is separate from spot quota. If your on-demand quota for that family is zero, the fallback fails. Check quotas now, not during the incident:
az vm list-usage -l swedencentral -o tableor AWS Service Quotas for "Running On-Demand G and VT instances".
Step 4: bring model pods up faster
Pods need nodes, images and weights. Speed it up:
- Confirm images are cached or pull from a registry in the same region.
- Lower replica count to what you can host, scale up as capacity appears.
- Make sure the deployment tolerates the new pool's taints and the node selector matches its labels.
kubectl -n llm scale deploy/llama-70b --replicas=2
kubectl -n llm rollout status deploy/llama-70b --timeout=20m
Step 5: confirm and fail back
- Model pods Ready;
/healthOK; gateway health checks green. - Load test briefly at normal traffic, then move traffic back from the fallback gradually (10%, 50%, 100%).
- Watch latency and KV cache metrics as traffic returns.
Step 6: redesign so spot cannot take everything
| Practice | Effect |
|---|---|
| Baseline on-demand / reserved capacity for the minimum replicas | Service survives losing all spot nodes |
| Spot only for the burst above baseline | Cheap scale-out, bounded risk |
| Diversify: several GPU sizes, families and zones | Evictions rarely hit all pools at once |
| Pod topology spread across zones and capacity types | Replicas not all on spot in one zone |
| PodDisruptionBudget + graceful termination | Clean handling of interruption notices |
| Interruption handling (Karpenter native, AWS Node Termination Handler, AKS eviction events) | Drains before reclaim where notice is given |
| Tested fallback route at the gateway | Users see degraded service, not errors |
Topology spread by capacity type (label names differ by platform):
topologySpreadConstraints:
- maxSkew: 1
topologyKey: karpenter.sh/capacity-type # AKS: kubernetes.azure.com/scalesetpriority
whenUnsatisfiable: ScheduleAnyway
labelSelector: { matchLabels: { app: llama-70b } }
Prevent it next time
- Minimum replicas always on non-spot capacity; document the cost of that baseline.
- Quotas for on-demand GPUs in two regions, checked monthly.
- Game day: cordon and drain all spot GPU nodes, measure time to recovery and whether fallback works.
- Alerts on GPU node count dropping below the baseline and on Pending model pods.