GPU inference node out of disk: model caches, huge images and ephemeral-storage evictions

#Symptoms

• Inference pods show Evicted with The node was low on resource: ephemeral-storage.
• New pods stay Pending on GPU nodes with the taint node.kubernetes.io/disk-pressure.
• Model downloads fail part way with No space left on device (OSError: [Errno 28]).
• Image pulls fail or take forever because kubelet keeps garbage-collecting and re-pulling the same 15 GB image.

GPU nodes fill up faster than normal nodes. Each model revision is tens of GB, serving images with CUDA libraries are often 10 GB or more, and every pod that downloads weights into emptyDir writes them again.

#Step 1: Confirm it's disk and find the node

kubectl get nodes -o custom-columns=NAME:.metadata.name,DISK:.status.conditions[?(@.type==\"DiskPressure\")].status
kubectl get pods -A --field-selector=status.phase=Failed -o wide | grep -i evicted
kubectl describe node <gpu-node> | grep -A6 -E "Conditions|Taints|ephemeral-storage"

Check what kubelet thinks is left:

kubectl get --raw "/api/v1/nodes/<gpu-node>/proxy/stats/summary" \
  | jq '{nodefs: .node.fs | {availableBytes, capacityBytes},
         imagefs: .node.runtime.imageFs | {availableBytes, capacityBytes}}'

nodefs holds pod logs, emptyDir volumes and writable layers. imagefs holds container images. On many nodes they are the same disk.

#Step 2: Find what uses the space

Get a shell on the node (kubectl debug node/<gpu-node> -it --image=ubuntu, then chroot /host), or use SSH or SSM:

df -h / /var/lib/containerd /var/lib/kubelet
du -xh --max-depth=2 /var/lib/kubelet/pods 2>/dev/null | sort -rh | head -15
du -xh --max-depth=1 /var/lib/containerd 2>/dev/null | sort -rh | head
crictl images | sort -k4 -h | tail -10

Then list per-pod ephemeral usage from the cluster side:

kubectl get --raw "/api/v1/nodes/<gpu-node>/proxy/stats/summary" \
  | jq -r '.pods[] | [.podRef.namespace, .podRef.name, (."ephemeral-storage".usedBytes/1e9|floor)] | @tsv' \
  | sort -k3 -n -r | head

#Step 3: Free space quickly

Restore service first, then clean up properly.

1. Delete evicted pod objects. They don't hold disk, but they clutter triage:

   kubectl delete pods -A --field-selector=status.phase=Failed
   

2. Remove unused images. This only removes images that no container is using:

   crictl rmi --prune
   

3. Restart the worst emptyDir consumer if it is a replica you can lose. Its emptyDir is deleted with the pod. Check the PodDisruptionBudget and remaining capacity first:

   kubectl get pdb -n <ns>
   kubectl delete pod <pod> -n <ns>
   

Caution

Don't delete files under /var/lib/containerd or /var/lib/kubelet by hand. You can corrupt the runtime's metadata and lose the node. Use crictl, delete pods, or drain and replace the node.

If space doesn't come back, cordon and drain the node and let the autoscaler or node pool replace it. On managed node groups a fresh node is often faster than cleaning one up:

kubectl cordon <gpu-node>
kubectl drain <gpu-node> --ignore-daemonsets --delete-emptydir-data --timeout=10m

#Step 4: Clean a shared Hugging Face cache

If models are cached on a hostPath or a shared volume (HF_HOME or HF_HUB_CACHE), old revisions pile up every time someone changes --revision or a repo is updated.

Recent huggingface_hub releases ship the hf CLI. Older releases use huggingface-cli scan-cache and delete-cache, so check hf --help on your version.

export HF_HUB_CACHE=/models/hf/hub   # wherever your servers point
hf cache ls                          # repos and their size
hf cache prune                       # remove detached revisions no branch or tag points to
hf cache rm model/<org>/<old-model> -y

A version-independent option is the Python API, which also gives you a dry run:

from huggingface_hub import scan_cache_dir

info = scan_cache_dir("/models/hf/hub")
print(f"total {info.size_on_disk/1e9:.1f} GB")
for repo in sorted(info.repos, key=lambda r: r.size_on_disk, reverse=True)[:10]:
    print(f"{repo.size_on_disk/1e9:7.1f} GB  {repo.repo_id}  revisions={len(repo.revisions)}")

# Keep only revisions that a ref (e.g. main) points to
old = [rev.commit_hash for repo in info.repos for rev in repo.revisions if not rev.refs]
strategy = info.delete_revisions(*old)
print("would free", strategy.expected_freed_size_str)
# strategy.execute()   # uncomment after checking nothing running uses these revisions

Warning

A model server that pinned a commit hash (not a branch) is using a revision with no ref. Check the --revision of every running deployment before you delete detached revisions, or the next restart will download the model again or fail if the node is offline.

#Step 5: Fix the pod spec

Most repeat incidents come from pods that write models to node disk with no limit.

spec:
  containers:
    - name: vllm
      image: vllm/vllm-openai:<pinned-tag>
      env:
        - name: HF_HOME
          value: /models/hf
      resources:
        requests:
          ephemeral-storage: "20Gi"   # logs, temp files, compile caches
        limits:
          ephemeral-storage: "40Gi"
      volumeMounts:
        - name: models
          mountPath: /models
  volumes:
    - name: models
      persistentVolumeClaim:
        claimName: model-cache        # ReadOnlyMany or ReadWriteMany if shared

With ephemeral-storage limits set, kubelet evicts the single pod that goes over its limit instead of putting the whole node under DiskPressure.

#Step 6: Tune node disk and image garbage collection

• Make GPU node pools' OS disks larger than general-purpose pools. Size them for at least two serving images plus the largest model you cache locally, with room to spare.
• If you can configure kubelet, check imageGCHighThresholdPercent and imageGCLowThresholdPercent, and the evictionHard value nodefs.available. Managed services expose these differently or not at all; check your platform's node configuration options.
• Pin and reuse serving image tags across deployments. Five teams using five patch versions of the same 12 GB image means 60 GB of images per node.

#Prevent it next time

• Alert on node disk before kubelet does. In Prometheus with node-exporter:

  (node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|overlay"}
    / node_filesystem_size_bytes{mountpoint="/",fstype!~"tmpfs|overlay"}) < 0.20
  

  Pair it with an alert on kube_node_status_condition{condition="DiskPressure",status="true"} == 1 from kube-state-metrics.

• Run a scheduled Job that prunes detached Hugging Face revisions on shared caches and reports the size of the cache.

• Require ephemeral-storage requests and limits on GPU workloads through admission policy (Kyverno, Gatekeeper or Azure Policy for AKS).

• Pre-pull the standard serving image onto new GPU nodes with a DaemonSet so the first scale-out doesn't start with a 10-minute pull.