Applies toAzure (GPU VM families, AKS node pools), AWS (EC2 GPU instances, EKS), OCI GPU shapes; self-hosted LLM inference and training clusters
Symptoms
- Azure:
AllocationFailed,ZonalAllocationFailed,SkuNotAvailable, orOperationNotAllowed ... exceeding approved ... quota. - AWS:
InsufficientInstanceCapacity,VcpuLimitExceeded, Auto Scaling activity "Could not launch". - Kubernetes: GPU pods Pending; autoscaler logs show scale-up failures.
Step 1: capacity or quota?
| Message | Meaning | Speed of fix |
|---|---|---|
| Quota exceeded / VcpuLimitExceeded | Your subscription/account limit | Quota increase request (minutes to days) |
| AllocationFailed / InsufficientInstanceCapacity | Provider has no free hardware of that size in that zone/region right now | Change where or what you ask for |
az vm list-usage --location westeurope -o table | grep -i -E "NC|ND" # quota per family
aws service-quotas get-service-quota --service-code ec2 --quota-code L-DB2E81BA # Running On-Demand G and VT instances (check code for your family)
Step 2: get capacity now
In order of usual speed:
- Other zones in the same region: allow the node pool / ASG to use all zones (multi-zone node groups).
- Other GPU sizes that can run the model: e.g. more smaller GPUs with tensor parallelism, or a different generation. Keep a tested list of fallback SKUs per model.
- Other regions where you already have quota and the model artefacts: route part of the traffic there through the gateway.
- Spot / low-priority capacity for non-critical traffic (batch, internal users), freeing on-demand for production.
- Quota increase if the issue is quota.
AWS EKS managed node groups and Karpenter can list several instance types. Give them a set, not one:
# Karpenter NodePool excerpt
requirements:
- key: karpenter.k8s.aws/instance-family
operator: In
values: ["g6e", "g6", "p5", "p4d"]
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand"]
AKS: add a second GPU node pool with another VM size and the same labels/taints, so pods can land on either.
Step 3: reduce demand per request
While capacity is short:
- Lower
max_tokensand cap prompt length at the gateway. - Turn off non-essential AI features (feature flags) and batch jobs.
- Serve a quantized variant (FP8/INT4) that fits more replicas per GPU, if already validated.
- Route overflow to a managed API (Azure OpenAI / Bedrock) if your policies allow it.
Step 4: guarantee baseline capacity
| Provider | Option |
|---|---|
| Azure | On-demand capacity reservations for specific VM sizes and zones (pay whether used or not); reserved instances/savings plans for price only |
| AWS | On-Demand Capacity Reservations (ODCR); Capacity Blocks for ML for fixed-duration GPU blocks |
| OCI | Capacity reservations for GPU shapes |
Reserve the baseline (minimum replicas across zones) and burst with autoscaling. Reservations and discounts are different: a savings plan lowers price but doesn't guarantee a machine.
az capacity reservation group create -g rg-gpu -n crg-inference --zones 1 2
az capacity reservation create -g rg-gpu -c crg-inference -n cr-nc80-z1 --sku Standard_NC80adis_H100_v5 --capacity 2 --zone 1
(Check which GPU sizes support capacity reservations in your region.)
Step 5: after the incident
- Record which SKUs/zones failed and when. Patterns help with planning and with your account team.
- Load-test fallback SKUs so they're ready to use, not theoretical.
- Revisit the capacity plan: baseline reserved, burst estimated, regional spread.
Tip
Keep model artefacts (weights, container images) replicated in at least two regions. Moving traffic to another region is only fast if the model is already there.
Prevent it next time
- Multi-zone, multi-SKU node pools for inference.
- Capacity reservations for baseline; quota headroom for burst.
- Gateway routing ready for a second region or managed API overflow.
- Alerts on scale-up failures, not just on Pending pods.