nvidia-smi cheat sheet for GPU inference nodes: inventory, memory, processes, clocks, errors and topology

#When to reach for it

You have a GPU node that is slow, out of memory, or not serving, and you need answers in the next five minutes. nvidia-smi ships with the driver, so it is on every working GPU host and inside any container that the NVIDIA Container Toolkit has set up. Prometheus and DCGM give you history. nvidia-smi shows you what the GPU is doing right now.

Tip

Field names for --query-gpu change between driver releases. Run nvidia-smi --help-query-gpu on the node to see what that driver supports before you script against a field.

#Inventory: what is in this box

The "CUDA Version" in the nvidia-smi header is the highest CUDA runtime that driver supports. It is not the toolkit version installed in your container. If a framework says no GPU is available, compare the two (see the driver and CUDA mismatch post).

#Memory and utilisation

# One CSV line per GPU, refreshed every 5 seconds
nvidia-smi --query-gpu=index,utilization.gpu,utilization.memory,memory.used,memory.total,temperature.gpu,power.draw \
  --format=csv,noheader,nounits -l 5

For a rolling per-second view that is easy to read during an incident:

nvidia-smi dmon -s pucm -d 1      # p=power/temp, u=utilisation, c=clocks, m=memory

#Who is using the GPU

nvidia-smi --query-compute-apps=gpu_uuid,pid,process_name,used_memory --format=csv
nvidia-smi pmon -c 5              # per-process SM and memory utilisation, 5 samples

Inside a container, PIDs belong to the host PID namespace and may not match ps in the container. On a Kubernetes node, map the PID to a pod:

# On the node (or a debug pod with hostPID)
cat /proc/<pid>/cgroup | grep -o 'pod[0-9a-f_-]*'
kubectl get pods -A -o wide --field-selector spec.nodeName=<node>

To run nvidia-smi inside an inference pod:

kubectl exec -it -n <ns> <pod> -- nvidia-smi

If the command is missing or reports no devices inside the pod but works on the node, the pod did not get a GPU. Check resources.limits["nvidia.com/gpu"] and the device plugin.

#Clocks, power and throttling

A GPU that reports high utilisation but low throughput is often running at reduced clocks.

nvidia-smi -q -d PERFORMANCE      # shows active throttle / clock event reasons
nvidia-smi -q -d CLOCK,POWER
nvidia-smi --query-gpu=index,clocks.sm,clocks.max.sm,power.draw,power.limit,temperature.gpu --format=csv

On recent drivers this section is called "Clocks Event Reasons" and the query fields are clocks_event_reasons.*. Older drivers use "Clocks Throttle Reasons" and clocks_throttle_reasons.*. Check --help-query-gpu for your driver.

#Errors: ECC, row remapping and Xid

nvidia-smi -q -d ECC
nvidia-smi --query-gpu=index,ecc.errors.uncorrected.volatile.total,ecc.errors.corrected.volatile.total --format=csv
nvidia-smi -q -d ROW_REMAPPER     # Ampere and newer
nvidia-smi -q -d PAGE_RETIREMENT  # older architectures
dmesg -T | grep -i xid            # driver-reported GPU faults

#Topology and NVLink

For multi-GPU tensor parallel serving, check how GPUs connect before blaming NCCL:

nvidia-smi topo -m        # matrix: NV# = NVLink, PIX/PXB = PCIe switch, SYS = across CPU sockets
nvidia-smi nvlink -s      # per-link status and speed

SYS between GPUs in the same tensor parallel group means traffic crosses the CPU interconnect and will be slower. Pin the group to GPUs connected by NVLink or the same PCIe switch.

#MIG

nvidia-smi -i 0 --query-gpu=mig.mode.current,mig.mode.pending --format=csv
nvidia-smi mig -lgip      # available GPU instance profiles
nvidia-smi mig -lgi       # GPU instances created
nvidia-smi -L             # also lists MIG devices with their UUIDs

With MIG enabled, nvidia-smi cannot show per-instance utilisation. Use DCGM for that.

#Changing settings

These commands change GPU state. Run them only on a drained node or during a planned change.

Warning

--gpu-reset fails if any process is using the GPU, and it kills in-flight work. On Kubernetes, cordon and drain the node first, and stop nvidia-persistenced and DCGM exporters if they hold the device. Many cloud VMs do not support reset; reboot the VM instead.

Prefer nvidia-persistenced (the daemon) over -pm 1 where your distribution ships it. Persistence mode set with -pm does not survive a reboot.

#Quick triage sequence

1. nvidia-smi - do all GPUs appear, and is anything at 100% memory with no processes listed?
2. nvidia-smi --query-compute-apps=... - which process owns the memory?
3. nvidia-smi -q -d PERFORMANCE - is the card throttled?
4. nvidia-smi -q -d ECC,ROW_REMAPPER plus dmesg | grep -i xid - is the hardware healthy?
5. nvidia-smi topo -m - for multi-GPU problems, is the topology what the serving config assumes?

#Prevent it next time

• Export the same fields continuously with DCGM Exporter so you have history, not just a snapshot.
• Alert on uncorrected ECC errors, pending row remaps and Xid events, and cordon the node automatically.
• Enable persistence on every GPU node through the image or the GPU Operator, not by hand.
• Keep a small script with your standard --query-gpu fields in the node image so everyone on call runs the same checks.