Introduction to GPU scheduling on Kubernetes for LLM inference: requests, taints, node selectors and sharing

#How Kubernetes sees GPUs

Kubernetes doesn't know about GPUs by itself. The NVIDIA device plugin (installed directly or by the NVIDIA GPU Operator, or preinstalled on some managed GPU node pools) advertises each GPU as an extended resource called nvidia.com/gpu on the node.

kubectl get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

Rules that differ from CPU and memory:

• GPUs are requested in whole units (1, 2, 4...). No 0.5.
• You set the GPU in limits. If you set requests too, it must equal limits.
• A GPU given to one container isn't shared with others (unless you configure sharing, below).

#Step 1: Request a GPU in a pod

A minimal vLLM deployment requesting one GPU:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-small
spec:
  replicas: 1
  selector:
    matchLabels: { app: vllm-small }
  template:
    metadata:
      labels: { app: vllm-small }
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.6.3        # pin a version you have tested
          args: ["--model", "Qwen/Qwen2.5-7B-Instruct", "--max-model-len", "8192"]
          ports: [{ containerPort: 8000 }]
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 32Gi
              cpu: "8"
            requests:
              memory: 32Gi
              cpu: "4"
          volumeMounts:
            - { name: shm, mountPath: /dev/shm }
      volumes:
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 8Gi }

The /dev/shm volume matters for inference servers that use shared memory (multi-GPU tensor parallel especially).

Check where it landed and that the GPU is visible:

kubectl get pod -l app=vllm-small -o wide
kubectl exec deploy/vllm-small -- nvidia-smi

#Step 2: Keep non-GPU pods off GPU nodes (taints)

GPU nodes are expensive. Without a taint, ordinary pods (log agents are fine, but web apps aren't) can fill their CPU and memory, and GPU pods can't schedule.

Taint the GPU node pool:

# AKS
az aks nodepool add -g rg-aks --cluster-name aks-prod -n gpu \
  --node-vm-size Standard_NC24ads_A100_v4 --node-count 1 \
  --node-taints sku=gpu:NoSchedule --labels gpu-type=a100

# Any cluster (per node)
kubectl taint nodes <gpu-node> nvidia.com/gpu=present:NoSchedule

Add a matching toleration to GPU workloads:

      tolerations:
        - key: "sku"
          operator: "Equal"
          value: "gpu"
          effect: "NoSchedule"

Daemonsets that must run on GPU nodes (device plugin, monitoring, log agents) need the toleration too. The GPU Operator's components include tolerations for common GPU taints; check yours matches.

#Step 3: Pick the GPU type (labels and node selectors)

Clusters often have several GPU types. Target one with a node label:

      nodeSelector:
        gpu-type: a100

If you run NVIDIA GPU Feature Discovery (part of the GPU Operator), nodes get labels like nvidia.com/gpu.product and nvidia.com/gpu.memory automatically:

kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/gpu.memory

      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: nvidia.com/gpu.memory
                    operator: Gt
                    values: ["40000"]     # MiB

Tip

Size the model to the GPU's memory before scheduling. A 7B model in FP16 needs roughly 14-16 GB for weights plus KV cache; a 70B model needs multiple GPUs or quantisation. Picking the node type by GPU memory avoids pods that schedule fine and then crash with CUDA out-of-memory.

#Step 4: Sharing a GPU between pods

Small models and dev workloads often don't need a whole GPU. Options:

With MIG, resources look like nvidia.com/mig-1g.10gb: 1 (names depend on GPU and MIG strategy). Time-slicing is enabled with a device plugin ConfigMap; see the NVIDIA device plugin docs for the exact format for your version.

Warning

Time-slicing gives no memory isolation. If two pods together need more GPU memory than exists, one of them crashes with out-of-memory. Don't time-slice production inference unless you control memory use per pod.

#Step 5: Scaling GPU pools

• Use the cluster autoscaler with a separate GPU node pool that can scale to zero (min count 0) for bursty workloads.
• GPU nodes can take several minutes to become ready (driver load, image pull of multi-GB images). Pre-pull images or use a small warm pool for latency-sensitive services.
• Scale inference pods on a request queue or GPU utilisation metric (DCGM exporter) rather than CPU.

# Quick view of GPU usage per pod if DCGM exporter + Prometheus are installed
# PromQL: avg by (pod) (DCGM_FI_DEV_GPU_UTIL)

#Quick troubleshooting

#Prevent problems later

• Taint every GPU pool and add tolerations only to workloads that need GPUs.
• Label pools by GPU type and memory, and select on them.
• Monitor GPU utilisation and memory per pod; idle GPUs are the biggest waste in AI platforms.
• Pin container image versions and driver versions together, and upgrade them as a tested pair.