Applies toKubernetes (AKS, EKS, OKE, self-managed) with the NVIDIA GPU Operator/device plugin; MIG-capable GPUs (e.g. A100, H100/H200 class); DCGM exporter for metrics
Measure first
With the DCGM exporter in Prometheus:
| Metric | Meaning |
|---|---|
DCGM_FI_DEV_GPU_UTIL | % time the GPU was busy |
DCGM_FI_DEV_FB_USED | Memory used |
DCGM_FI_PROF_SM_ACTIVE | Share of streaming multiprocessors active (better than GPU_UTIL) |
avg_over_time(DCGM_FI_PROF_SM_ACTIVE[1d])
Workloads with low SM activity and small memory use are candidates for sharing.
Options
| Option | Isolation | Fits |
|---|---|---|
| MIG (Multi-Instance GPU) | Hardware partitions with own memory and compute | Production small models, embedding services, multi-tenant |
| Time-slicing | None (shared memory, context switching) | Dev/test, notebooks, bursty light jobs |
| MPS (Multi-Process Service) | Partial | Many small inference processes from trusted workloads |
| Dedicated GPU | Full | Large models, latency-critical serving, training |
MIG with the GPU Operator
# Label a node to apply a MIG profile set (GPU Operator mig-manager)
# kubectl label node <gpu-node> nvidia.com/mig.config=all-1g.10gb --overwrite
apiVersion: v1
kind: Pod
metadata:
name: embedder
spec:
containers:
- name: tei
image: ghcr.io/huggingface/text-embeddings-inference:<version>
resources:
limits:
nvidia.com/mig-1g.10gb: 1
Available profiles depend on the GPU model; check nvidia-smi mig -lgip. Managed Kubernetes services may offer MIG settings on node pools.
Time-slicing
Configure the device plugin to advertise each GPU as several replicas:
version: v1
sharing:
timeSlicing:
resources:
- name: nvidia.com/gpu
replicas: 4
Pods then request nvidia.com/gpu: 1 and share the device. Memory isn't limited per pod, so one pod can starve others.
Packing and autoscaling
- Separate node pools: shared/MIG pool for small workloads, dedicated pool for large models.
- Cluster autoscaler or Karpenter scales GPU pools to zero when idle (dev/test).
- Taints and tolerations keep non-GPU pods off GPU nodes.
Warning
Time-slicing gives no memory isolation. Don't mix production inference with untrusted or experimental notebooks on the same time-sliced GPU.
Checklist
- GPU utilisation and memory measured per workload
- MIG for isolated small production workloads
- Time-slicing only for dev/test or trusted light jobs
- Separate node pools for shared and dedicated GPUs
- Scale-to-zero for idle GPU pools
- Non-GPU pods kept off GPU nodes