Sev1: GPU Xid errors and 'GPU has fallen off the bus'. Triage for inference and training nodes

Symptoms

nvidia-smi prints Unable to determine the device handle for GPU ... Unknown Error or hangs.
Pods on one node fail with CUDA error: an illegal memory access, CUDA error: unspecified launch failure or NCCL error.
Kernel log shows NVRM: Xid (PCI:0000:3b:00): 79, ... GPU has fallen off the bus.

sudo dmesg -T | grep -i xid
sudo journalctl -k --since "2 hours ago" | grep -i -E "xid|nvrm"
nvidia-smi -q -d ECC,PAGE_RETIREMENT,ROW_REMAPPER | less

Xid	Usually means	First action
13, 31, 43	Application error (bad kernel, illegal memory access, out-of-bounds)	Check app/driver/CUDA versions; usually not hardware
45	Preemptive cleanup after an earlier error	Look at the error before it
48	Double-bit ECC error	Drain; reset; replace if repeated
63, 64	ECC page retirement / row remapping event	Reset GPU to apply remap; replace if remapping fails
74	NVLink error	Drain; reset; hardware check if repeated
79	GPU fallen off the bus (PCIe)	Drain; reboot node; hardware/host ticket
92, 94, 95	ECC errors (94 contained, 95 uncontained)	94: restart affected app; 95: drain, reset
119, 120	GSP firmware errors/timeouts	Driver update / reset; check known driver issues

Check NVIDIA's Xid documentation for the meaning on your driver version. Codes are added over time.

Rule of thumb: one Xid 13/31 on one pod is usually code; 79/48/95, or repeated errors on the same GPU, is hardware or platform.

Kubernetes:

kubectl cordon gpu-node-07
kubectl drain gpu-node-07 --ignore-daemonsets --delete-emptydir-data --timeout=10m

Inference replicas reschedule onto healthy nodes (if capacity allows). For training jobs, restart from the last checkpoint on healthy nodes.

sudo nvidia-smi --gpu-reset -i 3

If reset fails or the GPU isn't visible, reboot the node.
On cloud VMs, if errors return: redeploy/repair the VM to move to other hardware (Azure: Redeploy or report the issue for the GPU SKU; AWS: stop/start to move host; also check for scheduled events). On bare metal: run NVIDIA field diagnostics (dcgmi diag -r 3) and open a hardware case.

dcgmi diag -r 2     # quicker; -r 3 / -r 4 are longer and more thorough

Run a short GPU burn/diagnostic before uncordoning.
kubectl uncordon gpu-node-07.
Keep a record per node and GPU serial (nvidia-smi -q | grep -i serial): repeated offenders get replaced.

Run DCGM Exporter and alert on Xid events (DCGM_FI_DEV_XID_ERRORS), ECC counts and row-remap failures.
Use a node problem detector or the GPU operator's health checks to taint unhealthy GPU nodes automatically.
Managed node health features (e.g. node auto-repair in managed Kubernetes) help, but check they detect GPU faults, not just kubelet health.

Tip
Keep spare GPU capacity (N+1 nodes per model) for inference. Draining a node is only painless if the remaining nodes can take its load.