Home / AI & GenAI

AI & GenAITroubleshootingIntermediate

NVIDIA driver and CUDA version mismatch: GPU not detected in VMs and containers

Understand driver vs toolkit versions, fix 'CUDA driver version is insufficient' and NVML mismatch errors, and get GPUs working in Docker and Kubernetes.

Deepak Bhardwaj2 min read

Applies toLinux GPU VMs on Azure, AWS and OCI; Docker with NVIDIA Container Toolkit; Kubernetes with the NVIDIA device plugin or GPU Operator

Two versions, two meanings

nvidia-smi

The header shows the driver version and "CUDA Version: 12.x". That CUDA number is the highest CUDA runtime the driver supports, not what's installed.

nvcc --version        # the CUDA toolkit installed on the host (may not exist; not needed for containers)
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"

The rule: the CUDA runtime used by your app or container must be ≤ the version the driver supports (with some forward compatibility exceptions on data centre GPUs).

Errors and fixes

Error Cause Fix
CUDA driver version is insufficient for CUDA runtime version Container/app built for a newer CUDA than the driver supports Upgrade the driver, or use an image built for an older CUDA
Failed to initialize NVML: Driver/library version mismatch The driver package was upgraded but the old kernel module is still loaded Reboot (or unload and reload the nvidia modules)
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver No driver, or the module didn't build for the new kernel Reinstall the driver (DKMS) for the running kernel
torch.cuda.is_available() is False with a working nvidia-smi CPU-only PyTorch wheel installed Install the CUDA build of PyTorch

After kernel updates

Unattended upgrades that install a new kernel are a common cause of "GPU disappeared after reboot". Either hold the kernel, or make sure the driver uses DKMS:

dkms status
sudo apt-mark hold linux-image-$(uname -r) linux-headers-$(uname -r)   # Ubuntu, if you choose to pin

Cloud options: Azure NVIDIA GPU Driver Extension, AWS Deep Learning AMIs or the NVIDIA driver AMIs, OCI GPU images with drivers preinstalled.

Docker

# NVIDIA Container Toolkit installed and configured
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If that test works but your app image doesn't, the problem is inside the image (CUDA version or CPU-only libraries).

Kubernetes

kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu
kubectl -n gpu-operator get pods              # with the NVIDIA GPU Operator
  • Allocatable GPU is 0 or missing → device plugin isn't running on that node, or the driver isn't loaded.
  • Pods must request nvidia.com/gpu: 1 in resources.limits.
  • AKS, EKS and OKE have GPU node images or add-ons that install drivers. Don't install the driver twice (node image and GPU Operator).

Tip

Record the driver version per node pool and test new container images against it in CI. "Works on my workstation" usually means a newer driver there.

Related fixes