AKS pods OOMKilled with exit code 137: container limit, node pressure or runtime heap?

#Symptom

Pods restart repeatedly, and kubectl get pods shows a rising RESTARTS count with CrashLoopBackOff or OOMKilled. Raising the memory limit helped for a day, then it came back. Before you raise it again, find out which of these three problems you have.

#Step 1: read the termination record

kubectl describe pod <pod> -n <ns> | grep -A8 "Last State"

kubectl get pod <pod> -n <ns> -o jsonpath=\
'{range .status.containerStatuses[*]}{.name}{" "}{.restartCount}{" "}{.lastState.terminated.reason}{" "}{.lastState.terminated.exitCode}{"\n"}{end}'

Exit code 137 means 128 + 9: the process received SIGKILL. Only the reason field tells you it was the kernel OOM killer. In a multi-container pod, check which container died; it's often a sidecar.

#Step 2: check for node-level pressure

kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp | tail -20
kubectl describe node <node> | sed -n '/Conditions:/,/Addresses:/p'
kubectl describe node <node> | sed -n '/Allocated resources:/,/Events:/p'
kubectl top node

If memory limits add up to well over 100% of allocatable while requests look modest, the node is overcommitted. Several pods can grow at once, and pods using more than their request are evicted first. Unexpected kernel kills also appear as node events with reason OOMKilling.

With Container Insights:

KubeEvents
| where TimeGenerated > ago(24h)
| where Reason in ("OOMKilling", "Evicted", "BackOff")
| summarize Count = count() by Reason, Namespace, Name, Computer
| order by Count desc

#Step 3: look at memory from inside the container

On cgroup v2 nodes (current AKS Linux images), these files show what the kernel enforces:

kubectl exec -n <ns> <pod> -c <container> -- sh -c '
  echo "limit:   $(cat /sys/fs/cgroup/memory.max)"
  echo "current: $(cat /sys/fs/cgroup/memory.current)"
  cat /sys/fs/cgroup/memory.events
  grep -E "^(anon|file|shmem) " /sys/fs/cgroup/memory.stat'

• oom_kill in memory.events counts kills in this cgroup.
• A high anon value means heap, stacks and native allocations. This is real application memory.
• A high file value is mostly page cache, which can usually be reclaimed. A big shmem value isn't reclaimable, and it includes emptyDir volumes with medium: Memory.

Kubernetes evaluates eviction on working set, so kubectl top pod --containers and the graph below show the number that matters:

Perf
| where TimeGenerated > ago(7d)
| where ObjectName == "K8SContainer" and CounterName == "memoryWorkingSetBytes"
| where InstanceName has "<workload-name>"
| summarize P50_MB = percentile(CounterValue, 50) / 1048576,
            P99_MB = percentile(CounterValue, 99) / 1048576,
            Max_MB = max(CounterValue) / 1048576
  by bin(TimeGenerated, 1h), InstanceName
| order by TimeGenerated asc

A steady climb with sawtooth drops at each restart points to a leak. A flat line with sudden spikes points to a large request, batch job or startup burst.

#Step 4: check that the runtime knows its limit

Older runtimes don't read cgroup v2 limits correctly and size their heap from node memory. That's a frequent cause after a node image upgrade moves the cluster to cgroup v2.

Verify what the JVM sees:

kubectl exec -n <ns> <pod> -- java -XX:+PrintFlagsFinal -version | grep -E "MaxHeapSize|UseContainerSupport"

Leave room for non-heap memory. Metaspace, threads, direct buffers and JNI allocations all count toward the limit, so a heap of 100% of the limit will always be killed.

#Step 5: exit 137 without OOMKilled

The kubelet sends SIGKILL if a container ignores SIGTERM for longer than terminationGracePeriodSeconds. Failing liveness probes restart the container in the same way. Check the events:

kubectl get events -n <ns> --field-selector involvedObject.name=<pod> --sort-by=.lastTimestamp

Look for Unhealthy and Killing. A memory-starved JVM stuck in garbage collection often fails liveness probes first, so one problem can look like the other.

#The fix

Set the memory request equal to the limit for important services. That removes overcommit for this container and makes it a much later eviction target:

resources:
  requests:
    cpu: "500m"
    memory: "2Gi"
  limits:
    memory: "2Gi"

Size it from the P99 working set plus a margin, not from a guess. To get a recommendation without automatic changes, enable the Vertical Pod Autoscaler in recommendation mode:

az aks update --name <cluster> --resource-group <rg> --enable-vpa

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"

Read the result with kubectl describe vpa api-vpa.

Warning

Raising a limit on an overcommitted node moves the problem from one pod to the whole node. Check Allocated resources and add capacity if the sum of requests no longer fits.

#Prevent it next time

• Alert on restarts with reason OOMKilled and on working set above 85% of the limit.
• Use a LimitRange so namespaces can't deploy containers without memory requests.
• Capture a heap dump on OOM (-XX:+HeapDumpOnOutOfMemoryError for Java) to a mounted volume.
• Related: node disk and eviction pressure and the AKS day-2 cheat sheet.