Sev1: AKS pods evicted across the cluster. Node DiskPressure from container logs and images

#Symptoms

• Many pods in Evicted or ContainerStatusUnknown state, then rescheduled, then evicted again.
• kubectl describe node shows DiskPressure=True and taint node.kubernetes.io/disk-pressure:NoSchedule.
• Pod events: The node was low on resource: ephemeral-storage. Container X was using 18Gi, which exceeds its request of 0.

#Step 1: scope it (2 minutes)

kubectl get nodes -o custom-columns=NAME:.metadata.name,DISK:.status.conditions[?\(@.type==\"DiskPressure\"\)].status
kubectl get pods -A --field-selector=status.phase=Failed | grep -c Evicted
kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp | tail -20

The eviction messages name the container that used the most ephemeral storage. That is your first suspect.

#Step 2: free space now

Delete evicted pod objects (they hold no resources, but clutter output and alerts):

kubectl get pods -A --field-selector=status.phase=Failed -o json \
  | jq -r '.items[] | select(.status.reason=="Evicted") | "\(.metadata.namespace) \(.metadata.name)"' \
  | xargs -n2 sh -c 'kubectl delete pod -n $0 $1'

Look inside a node with a debug pod:

kubectl debug node/aks-apps-12345678-vmss000003 -it --image=mcr.microsoft.com/cbl-mariner/busybox:2.0
# inside, the node filesystem is at /host
chroot /host
df -h /var
du -xh --max-depth=2 /var/lib/containerd /var/log/pods 2>/dev/null | sort -h | tail -15
crictl images | head; crictl rmi --prune

Warning

Do not delete files under /var/lib/containerd by hand. Use crictl rmi --prune for unused images and let the kubelet remove dead containers.

If a node cannot recover, cordon, drain and let AKS replace it:

kubectl cordon aks-apps-12345678-vmss000003
kubectl drain aks-apps-12345678-vmss000003 --ignore-daemonsets --delete-emptydir-data --timeout=10m
az aks nodepool delete-machines -g rg-aks --cluster-name aks-prod -n apps --machine-names aks-apps-12345678-vmss000003

delete-machines needs a recent CLI; otherwise scale the pool or reimage the instance with az vmss reimage.

#Step 3: find the cause

Find pods with the largest ephemeral usage via the kubelet summary API:

kubectl get --raw "/api/v1/nodes/aks-apps-12345678-vmss000003/proxy/stats/summary" \
  | jq -r '.pods[] | [.podRef.namespace, .podRef.name, (.["ephemeral-storage"].usedBytes/1073741824|floor)] | @tsv' \
  | sort -k3 -n | tail

#Step 4: contain it with limits

resources:
  requests:
    ephemeral-storage: "2Gi"
  limits:
    ephemeral-storage: "4Gi"
volumes:
- name: scratch
  emptyDir:
    sizeLimit: 2Gi

With a limit, only the offending pod is evicted instead of everything on the node. Add a namespace default with a LimitRange:

apiVersion: v1
kind: LimitRange
metadata: { name: ephemeral-defaults, namespace: shop }
spec:
  limits:
  - type: Container
    defaultRequest: { ephemeral-storage: 1Gi }
    default: { ephemeral-storage: 4Gi }

#Step 5: node pool sizing

az aks nodepool show -g rg-aks --cluster-name aks-prod -n apps --query "{osDisk:osDiskSizeGb, type:osDiskType, vm:vmSize}" -o table

A new pool with a larger or ephemeral OS disk, then move workloads and delete the old pool, is the clean path. Kubelet log rotation and image GC thresholds can be set through custom node configuration (--kubelet-config) on new pools.

#Step 6: confirm

• No node with DiskPressure=True for an hour.
• Eviction events stopped.
• Node disk usage alert below 75%:

InsightsMetrics
| where Namespace == "container.azm.ms/disk" and Name == "used_percent"
| summarize max(Val) by Computer, bin(TimeGenerated, 5m)

#Prevent it next time

• Require ephemeral-storage requests/limits in production namespaces (Azure Policy for AKS).
• Alert on node disk used above 80%.
• Logging: structured, at INFO, shipped by the agent; never debug level in prod by default.