EKS managed node group update stuck or failed with PodEvictionFailure: PDBs, surge capacity and when to use --force

#Symptom

You started update-nodegroup-version (for a new AMI release, a Kubernetes minor version or a launch template version). The node group stays in UPDATING, old nodes sit cordoned, and in the end the update comes back Failed with one of these errors:

• PodEvictionFailure: "Reached max retries while trying to evict pods from nodes in node group"
• NodeCreationFailure or AsgInstanceLaunchFailures: replacement nodes never joined

When the update fails, the node group is left half-done. Some nodes run the new AMI and some run the old one, and the cordoned old nodes hold back capacity.

#How the update works

Knowing the phases tells you where it got stuck:

Two details cause most of the trouble. Eviction goes through the Eviction API, so PodDisruptionBudgets are respected. And if pods are still on the node after 15 minutes and you didn't set --force, the update fails.

#Step 1: read the update record

CLUSTER=prod-eu1
NG=general-a

aws eks list-updates --name "$CLUSTER" --nodegroup-name "$NG" --query "updateIds" --output text
aws eks describe-update --name "$CLUSTER" --nodegroup-name "$NG" --update-id <update-id> \
  --query "update.{Status:status,Type:type,Errors:errors,Params:params}"

aws eks describe-nodegroup --cluster-name "$CLUSTER" --nodegroup-name "$NG" \
  --query "nodegroup.{Status:status,Release:releaseVersion,Version:version,Scaling:scalingConfig,Update:updateConfig,Health:health.issues}"

The errors[].resourceIds field names the nodes or instances involved. Write them down before you change anything.

#Step 2: find what blocks eviction

List the PDBs that allow no disruptions right now:

kubectl get pdb -A -o json | jq -r '
  .items[] | select(.status.disruptionsAllowed == 0)
  | "\(.metadata.namespace)/\(.metadata.name)  minAvailable=\(.spec.minAvailable // "-") maxUnavailable=\(.spec.maxUnavailable // "-") healthy=\(.status.currentHealthy)/\(.status.expectedPods)"'

Then list the pods still running on the cordoned old nodes:

for n in $(kubectl get nodes -l eks.amazonaws.com/nodegroup=$NG \
            --field-selector spec.unschedulable=true -o name); do
  echo "== $n"
  kubectl get pods -A --field-selector spec.nodeName=${n#node/} \
    -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,STATUS:.status.phase
done

The usual blockers:

Tip

The last two failures show up in the eviction response, not in the PDB list. Try evicting a single pod with kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --dry-run=server, or watch kubectl get events -A --field-selector reason=FailedEviction.

#Step 3: fix the cause, then restart the update

You can't resume a failed update. You start a new one with the same target. Check afterwards with kubectl get nodes -L eks.amazonaws.com/nodegroup-image (or the node's kubelet version) that every node ended up on the target.

Fix the PDB or the workload first. For example, scale a single-replica service temporarily:

kubectl -n payments scale deploy/ledger-api --replicas=2
kubectl -n payments rollout status deploy/ledger-api

Then raise the update parallelism if the node group is large, and start the update again:

aws eks update-nodegroup-config --cluster-name "$CLUSTER" --nodegroup-name "$NG" \
  --update-config maxUnavailablePercentage=20

aws eks update-nodegroup-version --cluster-name "$CLUSTER" --nodegroup-name "$NG" \
  --release-version <target-release>      # or --launch-template version=<n>,id=<lt-id>

maxUnavailable (a count) and maxUnavailablePercentage are mutually exclusive. Larger values finish sooner, but they evict more pods at once and need more surge capacity in each AZ.

#Step 4: NodeCreationFailure during scale-up

If the replacement nodes never become Ready, the problem has nothing to do with eviction:

ASG=$(aws eks describe-nodegroup --cluster-name "$CLUSTER" --nodegroup-name "$NG" \
  --query "nodegroup.resources.autoScalingGroups[0].name" --output text)
aws autoscaling describe-scaling-activities --auto-scaling-group-name "$ASG" --max-items 10 \
  --query "Activities[].{Status:StatusCode,Msg:StatusMessage}"

Common causes are a custom launch template AMI with broken bootstrap user data, insufficient capacity for the instance type, subnets out of IP addresses, an EC2 vCPU quota, or a node IAM role that has lost its access entry. Fix the cause, then run the update again.

#Step 5: when to use --force

aws eks update-nodegroup-version --cluster-name "$CLUSTER" --nodegroup-name "$NG" --force
# eksctl equivalent
eksctl upgrade nodegroup --cluster "$CLUSTER" --name "$NG" --force-upgrade

--force ignores PDBs. Once the 15-minute eviction window passes, EKS terminates the node with the remaining pods still on it.

Caution

A forced update can take down every replica of a quorum-based StatefulSet, or the only replica of a service, at once. Use it only after you have checked that every pod still on the remaining old nodes can safely disappear. Use it for urgent AMI security patches only after you've tried scaling and fixing the PDBs.

#Prevent it next time

• Add a CI policy check (Kyverno, Gatekeeper or a linter) that rejects PDBs with minAvailable equal to the replica count, and single-replica workloads that have a PDB.
• Set unhealthyPodEvictionPolicy: AlwaysAllow on stateless workload PDBs so crash-looping pods can't block drains.
• Keep quorum-based stateful systems in their own node groups with maxUnavailable: 1, and roll them with an operator that knows how to rebalance.
• Before every update, run the disruptionsAllowed == 0 query and fail the pipeline if anything shows up.
• Check that each AZ can hold the surge nodes (instance quota, subnet IPs and capacity) before you start the rollout.
• If Cluster Autoscaler or another autoscaler manages the node group, check it doesn't scale down the new nodes while the update runs. Pause scale-down during the window if it does.