Kubernetes GPU node shows 0 allocatable nvidia.com/gpu: device plugin, drivers, container runtime and Xid errors

#Symptoms

• kubectl describe node <gpu-node> shows no nvidia.com/gpu under Capacity/Allocatable, or shows 0.
• GPU pods are Pending with 0/5 nodes are available: 5 Insufficient nvidia.com/gpu.
• It worked before a node image upgrade, driver upgrade or reboot.
• Allocatable shows fewer GPUs than the VM has (for example 7 of 8).

kubectl get nodes -o custom-columns='NAME:.metadata.name,CAP:.status.capacity.nvidia\.com/gpu,ALLOC:.status.allocatable.nvidia\.com/gpu'

#The chain that must work

GPU hardware ─► NVIDIA kernel driver ─► NVIDIA Container Toolkit ─► containerd runtime config ─► device plugin pod ─► kubelet advertises nvidia.com/gpu

Find the first broken link.

#Step 1: Is the device plugin running on that node?

kubectl get pods -A -o wide | grep -Ei 'nvidia-device-plugin|gpu-operator' | grep <gpu-node>

With the GPU Operator, the namespace is usually gpu-operator, and there are several daemonsets: driver, container-toolkit, device-plugin, dcgm-exporter, gpu-feature-discovery, validator.

kubectl -n gpu-operator get pods -o wide --field-selector spec.nodeName=<gpu-node>
kubectl -n gpu-operator logs <device-plugin-pod> --all-containers --tail=100

Check tolerations versus taints if the pod is missing:

kubectl get node <gpu-node> -o jsonpath='{.spec.taints}'; echo
kubectl -n gpu-operator get ds -o name | xargs -I{} kubectl -n gpu-operator get {} -o jsonpath='{.metadata.name}{" "}{.spec.template.spec.tolerations}{"\n"}'

#Step 2: Is the driver loaded on the host?

Open a debug shell on the node:

kubectl debug node/<gpu-node> -it --image=ubuntu:22.04
chroot /host
nvidia-smi
lsmod | grep nvidia
dmesg -T | grep -iE 'nvrm|nvidia|xid' | tail -30

Warning

Don't install drivers on the host and let the GPU Operator install them too. Two driver sources fight each other. On AKS and EKS GPU-optimised images, drivers are usually preinstalled; set the GPU Operator's driver.enabled=false in that case. Check your platform's guidance.

On AKS, GPU node pools install drivers by default unless you chose to skip GPU driver installation (to manage drivers with the Operator). Know which mode each pool uses.

#Step 3: Container toolkit and containerd runtime

The device plugin and workloads need the NVIDIA runtime to inject driver libraries into containers.

# On the node (after chroot /host)
cat /etc/containerd/config.toml | grep -A5 -i nvidia
nvidia-ctk --version 2>/dev/null

You should see a runtime named nvidia with BinaryName pointing at nvidia-container-runtime, and (depending on setup) either it's the default runtime or workloads use a RuntimeClass named nvidia.

kubectl get runtimeclass

If a nvidia RuntimeClass exists and the default runtime isn't nvidia, the device plugin and GPU pods need runtimeClassName: nvidia. The GPU Operator sets this up; mismatches happen after someone edits config.toml or a node image upgrade resets it.

After fixing the config:

systemctl restart containerd

Then delete the device plugin pod on that node so it restarts.

#Step 4: GPU Operator validator

The Operator runs a validator that tests driver, toolkit, CUDA and plugin in order:

kubectl -n gpu-operator logs -l app=nvidia-operator-validator --all-containers --tail=50
kubectl -n gpu-operator get pods -l app=nvidia-operator-validator -o wide

The failing init container name tells you which stage failed (driver-validation, toolkit-validation, cuda-validation, plugin-validation).

#Step 5: Hardware and Xid errors

If only some GPUs are missing, or GPUs disappear under load:

dmesg -T | grep -i xid
nvidia-smi -q | grep -iE 'retired|remapp|ecc' -A3

Xid errors are NVIDIA driver error codes. Some are application faults; others (for example "GPU has fallen off the bus" or repeated uncorrectable ECC errors) point to hardware. Look up the Xid number in NVIDIA's Xid documentation.

kubectl cordon <gpu-node>
kubectl drain <gpu-node> --ignore-daemonsets --delete-emptydir-data

On cloud GPU VMs, report repeated hardware Xids to the provider; they can mark the host as unhealthy.

#Step 6: MIG or time-slicing config changed the resource name

If MIG is enabled, the node may advertise nvidia.com/mig-1g.10gb (mixed strategy) instead of nvidia.com/gpu. If time-slicing is on, it may advertise more nvidia.com/gpu than physical GPUs, or a renamed resource (nvidia.com/gpu.shared), depending on config.

kubectl get node <gpu-node> -o json | jq '.status.allocatable | with_entries(select(.key|startswith("nvidia.com")))'

Pods requesting nvidia.com/gpu won't schedule on a node that only advertises MIG resources.

#Symptom → fix table

#Prevent it next time

• Pin node image, driver and GPU Operator versions together and upgrade them as a tested set on one pool first.
• Alert when a GPU node's allocatable nvidia.com/gpu is lower than its expected count.
• Run DCGM exporter and alert on Xid and ECC error metrics.
• Keep a runbook step to cordon and replace GPU nodes with hardware errors instead of debugging them in place.