Home / AI & GenAI

AI & GenAIHow-toIntermediate

NVIDIA GPU Operator on Kubernetes - installing it, checking it works and fixing pods that can't see the GPU

The NVIDIA GPU Operator installs the driver, container toolkit, device plugin and monitoring on GPU nodes. Install it with Helm, validate with a CUDA test pod and nvidia-smi, and fix the usual driver and plugin problems.

Deepak Bhardwaj4 min read

Applies toNVIDIA GPU Operator (Helm chart) on Kubernetes 1.28+ including AKS, EKS, OKE and on-premises clusters; NVIDIA data-centre GPUs

What the operator manages

Component Job
NVIDIA driver (containerised) Kernel driver on each GPU node
Container toolkit Lets the container runtime expose GPUs to containers
Device plugin Advertises nvidia.com/gpu as a schedulable resource
GPU Feature Discovery Labels nodes with GPU model, memory, MIG capability
DCGM exporter GPU metrics for Prometheus
MIG manager Applies MIG partitioning (on supported GPUs)
Validator Checks each layer and reports status

Some managed services install the driver or device plugin themselves. For example, AKS GPU node pools install the driver by default unless you opt out, and some EKS AMIs include drivers. In that case, disable those parts in the operator to avoid two drivers. Check your platform's GPU documentation.

1. Install with Helm

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

kubectl create namespace gpu-operator
kubectl label --overwrite namespace gpu-operator pod-security.kubernetes.io/enforce=privileged

helm install gpu-operator nvidia/gpu-operator -n gpu-operator --wait

Common values:

# driver already installed on the node image (managed node pools, pre-built AMIs)
helm install gpu-operator nvidia/gpu-operator -n gpu-operator --wait \
  --set driver.enabled=false

# driver and toolkit both pre-installed
  --set driver.enabled=false --set toolkit.enabled=false

Pin the chart version (--version) in production and upgrade on purpose. Check that the driver version the chart installs supports your GPU and CUDA version.

2. Check the rollout

kubectl get pods -n gpu-operator
kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/gpu.count
kubectl describe node <gpu-node> | grep -A3 -E "Capacity|Allocatable" | grep nvidia

You should see nvidia.com/gpu: <n> under Allocatable, and node labels showing the GPU model. All operator pods should be Running or Completed (the validator pods complete).

3. Test pod

apiVersion: v1
kind: Pod
metadata:
  name: cuda-test
spec:
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

kubectl apply -f cuda-test.yaml
kubectl logs cuda-test

The output should show the GPU model, driver version and the CUDA version the driver supports. Pick a CUDA image tag that your driver supports.

Troubleshooting

Symptom Likely cause Check / fix
No nvidia.com/gpu on the node Device plugin not running, or the driver failed kubectl logs -n gpu-operator -l app=nvidia-device-plugin-daemonset
Driver pod in CrashLoopBackOff Kernel headers missing, Secure Boot blocking the module, unsupported OS kubectl logs -n gpu-operator <nvidia-driver-daemonset-pod> -c nvidia-driver-ctr
Driver pod fails on a managed pool Driver already installed on the host Reinstall with driver.enabled=false
Pod Pending: Insufficient nvidia.com/gpu All GPUs allocated, or the pod lacks the taint toleration kubectl describe pod; check taints and other GPU pods
nvidia-smi not found in the container Container runtime not configured (toolkit) Check the toolkit pod logs; restart containerd after toolkit install if required
CUDA driver version is insufficient Image CUDA newer than the driver supports Use an older CUDA base image or upgrade the driver
GPUs disappear after node reboot Driver pod restarting; validator not finished Wait for the validator; check driver pod logs

Overall operator health:

kubectl get clusterpolicies.nvidia.com -o jsonpath='{.items[0].status.state}{"\n"}'

It should report ready.

Monitoring

The DCGM exporter exposes metrics such as DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED (memory used) and DCGM_FI_DEV_XID_ERRORS. Scrape it with Prometheus (a ServiceMonitor if you use the Prometheus Operator) and alert on:

  • XID errors (hardware or driver faults)
  • GPU memory near full for long periods
  • Allocated but idle GPUs (utilisation near zero), which is wasted cost

Node pool hygiene

  • Taint GPU nodes (nvidia.com/gpu=present:NoSchedule) so only GPU workloads land there.
  • Set GPU limits in every pod. Kubernetes GPUs are whole devices unless you use MIG or time-slicing.
  • Use separate node pools per GPU type, and node selectors on nvidia.com/gpu.product.

Checklist

  • Operator installed with the right driver/toolkit settings for the platform
  • Nodes show nvidia.com/gpu allocatable and product labels
  • CUDA test pod runs nvidia-smi
  • ClusterPolicy state is ready
  • DCGM metrics scraped and alerted
  • GPU nodes tainted; workloads tolerate and request GPUs

Related fixes