Checklist: upgrading NVIDIA GPU drivers on Kubernetes inference node pools without an outage

#Goal

Move every GPU node to a new NVIDIA driver while inference traffic keeps flowing. A driver change kills the GPU processes on each node. If you do it on too many nodes at once, or before checking compatibility, every model server goes down. This checklist covers the checks, the order of work and the rollback.

First, find out who owns the driver. That decides which mechanism you use:

# Is the GPU Operator managing the driver?
kubectl get clusterpolicies.nvidia.com -o jsonpath='{.items[0].spec.driver.enabled}{" "}{.items[0].spec.driver.version}{"\n"}'
kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -o wide

# Current driver on every GPU node
kubectl get nodes -l nvidia.com/gpu.present=true \
  -o custom-columns=NODE:.metadata.name,DRIVER:.metadata.labels.nvidia\\.com/cuda\\.driver\\.major

The nvidia.com/cuda.driver.* and nvidia.com/gpu.present labels come from GPU Feature Discovery. If they are missing, run nvidia-smi on a node or in a driver pod instead.

Caution

Do not run two driver owners at once. On AKS, a node pool created without --gpu-driver none already has a driver installed, so the GPU Operator's driver must be disabled (driver.enabled=false) there. Two drivers on one node usually means no GPUs at all after the next reboot.

#1. Compatibility checks (before any change)

• Target driver branch supports every GPU model in the fleet (check NVIDIA's release notes for the branch)
• Target driver supports the CUDA version in each serving image. Newer drivers run older CUDA runtimes, but the reverse needs a compat package. List the images: kubectl get pods -A -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | sort -u | grep -iE 'vllm|triton|tgi|sglang'
• GPU Operator version supports the target driver (check the operator's platform support matrix)
• Kernel version on the node image has a matching precompiled driver, or the driver container can build against it
• Fabric Manager version matches the driver exactly on NVSwitch systems (HGX H100/A100 8-GPU)
• MIG layouts, time-slicing config and DCGM Exporter still work on the new branch (read the release notes for removed features)
• Change ticket raised and model owners told about the window

#2. Capacity and disruption controls

Each node you upgrade drops its GPU pods. You need enough spare capacity, and the model servers need to stay above their minimum replica count.

• Spare GPU capacity of at least one node per node pool, or surge enabled. Check quota before you rely on surge.
• Every model Deployment has more than one replica and a PodDisruptionBudget:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: llm-chat-pdb
  namespace: inference
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: llm-chat

• Readiness probes only pass once the model is loaded, so traffic isn't sent to a pod that is still loading weights
• Model weights are cached (PVC, node-local cache or pre-pulled image), so rescheduled pods are back in minutes, not 20+
• Upgrade window avoids peak traffic and any planned batch jobs

Warning

A PDB with minAvailable equal to the replica count blocks every drain, and the upgrade stalls on the first node. A deployment with one replica and no PDB is gone the moment its node drains. Check both before you start.

#3. Configure the upgrade (GPU Operator)

Set the upgrade controller to one node at a time, using drain behaviour that respects PDBs:

# ClusterPolicy excerpt: spec.driver.upgradePolicy
spec:
  driver:
    upgradePolicy:
      autoUpgrade: true
      maxParallelUpgrades: 1
      maxUnavailable: 25%
      waitForCompletion:
        timeoutSeconds: 0
        podSelector: ""
      gpuPodDeletion:
        force: false
        timeoutSeconds: 300
        deleteEmptyDir: false
      drain:
        enable: false
        force: false
        podSelector: ""
        timeoutSeconds: 300
        deleteEmptyDir: false

• maxParallelUpgrades: 1 for the first pool. Raise it only after the canary passes.
• waitForCompletion.podSelector set to your batch or fine-tuning jobs if they must finish rather than be evicted
• Exclude the nodes you are not ready to touch:

kubectl label node <node> nvidia.com/gpu-driver-upgrade.skip=true

#4. Canary one node

• Pick one node in the least critical pool and remove the skip label from it only
• Change the driver version:

kubectl patch clusterpolicies.nvidia.com/cluster-policy --type='json' \
  -p='[{"op": "replace", "path": "/spec/driver/version", "value":"<target-version>"}]'

• Watch the upgrade state label move through upgrade-required → cordon-required → ... → upgrade-done:

kubectl get nodes -L nvidia.com/gpu-driver-upgrade-state -w

For managed drivers, the canary is a small, separate node pool on the new image:

# AKS: roll the node image on one pool with surge
az aks nodepool upgrade -g <rg> --cluster-name <aks> -n gpucanary --node-image-only --max-surge 1
# EKS: move one managed node group to a newer AMI release
aws eks update-nodegroup-version --cluster-name <eks> --nodegroup-name gpu-canary --release-version <ami-release>

#5. Validate the canary

• Driver version on the node: kubectl exec -n gpu-operator <driver-pod> -- nvidia-smi --query-gpu=driver_version --format=csv,noheader
• Node advertises the same GPU count as before: kubectl get node <node> -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'
• Operator validator pods completed: kubectl get pods -n gpu-operator -l app=nvidia-operator-validator
• A model pod scheduled on the node becomes Ready and answers a test request
• Latency and tokens per second match the old nodes (compare in your dashboard for 30-60 minutes)
• No new Xid lines: kubectl debug node/<node> -it --image=busybox -- chroot /host dmesg -T | grep -i xid
• DCGM Exporter metrics still arrive for the node

#6. Roll out the rest

• Remove the skip label pool by pool, starting with the least critical
• Raise maxParallelUpgrades only if spare capacity covers it
• Keep watching error rate and p95 latency for each model during every pool
• Pause the rollout if any node reaches upgrade-failed or a model drops below its minimum replicas (re-add the skip label to the remaining nodes)

#7. Rollback plan (write it before you start)

• Operator-managed: patch /spec/driver/version back to the previous version. The same controller rolls the nodes back one at a time.
• Managed node images: keep the old node pool or node group until the new one has run cleanly for a full business day, then scale it down
• Previous driver version, image tags and AMI or node image versions recorded in the change ticket

#Prevent it next time

• Pin the driver version explicitly in ClusterPolicy or the node image. Never follow "latest".
• Keep one canary GPU node pool permanently, and upgrade it a week ahead of production.
• Add a validation job (load a small model, run a fixed prompt, check tokens per second) that runs after every node upgrade.
• Review driver, CUDA, operator and Fabric Manager versions together each quarter, and record them in one place.