Monitoring GPU inference nodes with DCGM Exporter and Prometheus: the metrics and alerts that matter

#Why DCGM

nvidia-smi is fine for a quick look at one node. For a fleet of inference nodes you need metrics over time, per GPU and per pod, with alerts. NVIDIA's DCGM Exporter reads metrics from the Data Center GPU Manager and exposes them in Prometheus format, by default on port 9400 at /metrics.

#Step 1: deploy it

#Kubernetes with GPU Operator

The NVIDIA GPU Operator deploys DCGM Exporter by default. Check it is running:

kubectl get pods -n gpu-operator -l app=nvidia-dcgm-exporter -o wide
kubectl get svc -n gpu-operator | grep dcgm

#Kubernetes without the operator (Helm)

helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts
helm repo update
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter -n monitoring --create-namespace

Make sure it schedules only on GPU nodes (node selector or tolerations to match your GPU node taints).

#Standalone host (Docker)

docker run -d --gpus all --cap-add SYS_ADMIN --restart unless-stopped \
  -p 9400:9400 --name dcgm-exporter nvcr.io/nvidia/k8s/dcgm-exporter:<tag>

Pick the image tag that matches your driver and DCGM version from NVIDIA's NGC catalog.

Quick check:

curl -s localhost:9400/metrics | grep -E "^DCGM_FI_DEV_(GPU_UTIL|FB_USED)" | head

#Step 2: scrape with Prometheus

With the Prometheus Operator, a ServiceMonitor (the Helm chart can create one):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dcgm-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: dcgm-exporter
  namespaceSelector:
    any: true
  endpoints:
    - port: metrics
      interval: 15s

Label names differ between chart versions; match your service's labels (kubectl get svc -n <ns> --show-labels).

In Kubernetes, DCGM Exporter adds namespace, pod and container labels for GPUs assigned to pods, so you can see which workload is on which GPU.

#Step 3: the metrics that matter

Profiling (DCGM_FI_PROF_*) metrics depend on GPU model and the exporter's metric config file; enable them in the counters CSV if they don't appear.

Tip

Inference servers such as vLLM pre-allocate most GPU memory for the KV cache at startup. FB_USED at 90% is normal for them and not a leak. Watch the server's own KV-cache usage metric for real memory pressure, and use FB_USED to spot unexpected processes.

#Step 4: useful PromQL

# Average utilisation per node over 15 minutes
avg by (Hostname) (avg_over_time(DCGM_FI_DEV_GPU_UTIL[15m]))

# GPU memory used % per GPU
100 * DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)

# Which pod is on which GPU, with memory used
sum by (namespace, pod, gpu) (DCGM_FI_DEV_FB_USED)

# Idle GPUs: under 5% for 6 hours (expensive capacity doing nothing)
max_over_time(DCGM_FI_DEV_GPU_UTIL[6h]) < 5

#Step 5: alerts

groups:
- name: gpu
  rules:
  - alert: GpuXidError
    expr: DCGM_FI_DEV_XID_ERRORS > 0
    for: 1m
    labels: {severity: critical}
    annotations:
      summary: "XID {{ $value }} on {{ $labels.Hostname }} GPU {{ $labels.gpu }}"
  - alert: GpuHot
    expr: DCGM_FI_DEV_GPU_TEMP > 85
    for: 10m
    labels: {severity: warning}
  - alert: GpuMetricsMissing
    expr: absent(DCGM_FI_DEV_GPU_UTIL)
    for: 10m
    labels: {severity: warning}
  - alert: GpuIdleLong
    expr: max_over_time(DCGM_FI_DEV_GPU_UTIL[6h]) < 5
    labels: {severity: info}

Pick temperature thresholds from your GPU's specification; data centre GPUs throttle at different points.

#XID errors to know

Check NVIDIA's XID documentation for your driver; meanings are refined between releases.

#Troubleshooting the exporter

#Prevent it next time

• Treat DCGM Exporter as part of the GPU node image or operator install, not an add-on.
• Build one Grafana board per cluster: utilisation, memory, temperature, power, XID, plus inference latency from the serving layer.
• Route XID alerts to the platform team with a runbook to cordon and drain the node.
• Review idle-GPU alerts weekly; idle GPUs are the biggest waste in an inference platform.