Home / AI & GenAI

AI & GenAITroubleshootingIntermediate

Sev1: inference pods stuck Pending with 'Insufficient nvidia.com/gpu'

New model replicas never start and the scheduler says no node has GPUs, even though GPU nodes exist. Check whether GPUs are advertised, whether taints and selectors match, whether the device plugin or GPU operator is broken, and whether the node pool can scale at all.

Deepak Bhardwaj3 min read

Applies toKubernetes 1.28+ on AKS, EKS, OKE or on-prem with the NVIDIA device plugin or NVIDIA GPU Operator

Symptom

kubectl -n llm describe pod vllm-llama-7f9c-abcde
# Events:
#   Warning  FailedScheduling  0/6 nodes are available: 3 Insufficient nvidia.com/gpu,
#            3 node(s) didn't match Pod's node affinity/selector.

Step 1: do nodes advertise GPUs?

kubectl get nodes -L agentpool,node.kubernetes.io/instance-type \
  -o custom-columns='NAME:.metadata.name,GPU_ALLOC:.status.allocatable.nvidia\.com/gpu,GPU_CAP:.status.capacity.nvidia\.com/gpu'
Result Meaning
GPU nodes show <none> Device plugin not running/registered on them (Step 2)
Numbers shown but all in use Real capacity shortage (Step 4)
GPU nodes absent Node pool scaled to 0 or failed to scale (Step 4)

What's using GPUs now:

kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].resources.limits."nvidia.com/gpu" != null)
  | [.metadata.namespace, .metadata.name, .spec.nodeName, .status.phase] | @tsv'

Completed or stuck pods (e.g. failed jobs) may still hold GPUs.

Step 2: device plugin / GPU operator health

kubectl -n gpu-operator get pods          # or kube-system for a standalone device plugin
kubectl -n gpu-operator logs ds/nvidia-device-plugin-daemonset --tail=50
kubectl -n gpu-operator get pods -l app=nvidia-operator-validator

Common failures:

  • Driver not loaded after a node image upgrade (driver container failing, or pre-installed driver mismatch). Logs: could not load NVML library.
  • Node image already has drivers and the operator also tries to install them. Set driver.enabled=false in the operator values for images with pre-installed drivers (common on managed GPU node images).
  • Container toolkit not configured for the runtime (containerd config).

On a node:

kubectl debug node/gpu-node-01 -it --image=ubuntu -- chroot /host nvidia-smi

Step 3: scheduling constraints

GPU node pools are usually tainted. The pod needs the matching toleration and selector:

tolerations:
  - key: "nvidia.com/gpu"          # or sku=gpu:NoSchedule on AKS, depending on your pool setup
    operator: "Exists"
    effect: "NoSchedule"
nodeSelector:
  agentpool: gpua100
resources:
  limits:
    nvidia.com/gpu: 1

Check node taints:

kubectl describe node gpu-node-01 | grep -A3 Taints

Also check: MIG profiles (a pod requesting nvidia.com/gpu won't fit a node advertising nvidia.com/mig-1g.10gb), time-slicing config, and ResourceQuota limits in the namespace.

Step 4: capacity

  • Cluster autoscaler events: kubectl -n kube-system logs deploy/cluster-autoscaler | grep -i gpu (on AKS/EKS managed autoscaler, check the provider's logs/events).
  • Node pool at max count → raise the maximum.
  • Scale-up fails with allocation or capacity errors → regional GPU shortage or quota. See the GPU capacity runbook (other zones/regions, reservations, smaller GPUs).
  • Quota: check vCPU quota for the GPU VM family in the subscription/account.

Quick recovery options

Option Effect
Delete finished/stuck GPU pods Frees GPUs immediately
Lower replicas of non-critical GPU workloads (batch, dev) Frees GPUs for production inference
PriorityClass for production inference Scheduler preempts lower-priority GPU pods automatically
Serve a smaller/quantized model on available GPUs Fewer or smaller GPUs per replica 
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: inference-critical }
value: 100000
preemptionPolicy: PreemptLowerPriority

Tip

Separate GPU node pools for production inference and for experiments/training. Shared pools turn every big experiment into a production incident.

Prevent it next time

  • Alerts on Pending pods requesting GPUs for more than a few minutes.
  • GPU operator/device plugin version pinned and tested with each node image upgrade.
  • PriorityClasses and separate pools; quotas for non-production namespaces.
  • Capacity reservations for baseline inference.

Related fixes