Sev1: OKE node pool upgrade is stuck. Nodes cordoned, drains blocked and pods pending

#Symptoms

• kubectl get nodes shows several nodes Ready,SchedulingDisabled for a long time.
• Pods Pending with 0/N nodes are available.
• The node pool work request in the console is IN_PROGRESS for much longer than usual, or FAILED.

#Step 1: see the state (3 minutes)

kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase=Pending | head -20

oci ce work-request list --compartment-id $COMP --cluster-id $CLUSTER \
  --query 'data[0:5].{id:id, op:"operation-type", status:status, started:"time-started"}' --output table
oci ce work-request-error list --work-request-id $WR --compartment-id $COMP
oci ce work-request-log-entry list --work-request-id $WR --compartment-id $COMP --query 'data[-10:].message'

#Step 2: unblock drains

Drains stop on PodDisruptionBudgets that cannot be satisfied, and on pods with local storage or no controller.

kubectl get pdb -A
kubectl get pdb -A -o json | jq -r '.items[] | select(.status.disruptionsAllowed==0) | "\(.metadata.namespace)/\(.metadata.name) minAvailable=\(.spec.minAvailable) healthy=\(.status.currentHealthy)"'

A PDB with minAvailable equal to the replica count (or a single-replica Deployment with minAvailable: 1) blocks forever. Options, safest first:

1. Scale the workload up by one so the PDB allows a disruption: kubectl -n shop scale deploy/cart --replicas=3.
2. Temporarily relax the PDB: kubectl -n shop patch pdb cart-pdb --type merge -p '{"spec":{"minAvailable":1}}'.
3. Drain manually with a timeout, accepting disruption for that one workload.

kubectl drain 10.0.10.15 --ignore-daemonsets --delete-emptydir-data --timeout=10m

Caution

--disable-eviction or deleting pods by hand bypasses PDBs. Use it only for workloads you have confirmed can take the outage, and tell their owners.

#Step 3: new nodes do not become Ready

kubectl describe node <new-node> | sed -n '/Conditions/,/Addresses/p'
oci ce node-pool get --node-pool-id $NP --query 'data.nodes[].{name:name, state:"lifecycle-state", err:"node-error"}'

Common causes:

HIST=$(oci compute console-history capture --instance-id $NODE_INSTANCE --query 'data.id' --raw-output)
sleep 20
oci compute console-history get-content --instance-console-history-id $HIST --file - | tail -50

#Step 4: restore capacity now

If the workload is short of capacity, add capacity first, then finish the upgrade:

oci ce node-pool update --node-pool-id $NP --size 8      # scale the pool up
# or create a temporary pool at the target version
oci ce node-pool create --cluster-id $CLUSTER --compartment-id $COMP --name np-surge \
  --kubernetes-version v1.31.1 --node-shape VM.Standard.E5.Flex \
  --node-shape-config '{"ocpus":4,"memoryInGBs":32}' \
  --placement-configs '[{"availabilityDomain":"'"$AD"'","subnetId":"'"$WORKER_SUBNET"'"}]' \
  --size 3 --node-image-id $OKE_IMAGE

Out-of-capacity errors for a shape: try another AD or fault domain, or another shape. Service limit errors: check oci limits resource-availability get for the compute limit and request an increase.

Uncordon old nodes if you need their capacity back while you fix the new ones:

kubectl uncordon 10.0.10.15

#Step 5: finish the upgrade

Once drains are unblocked and new nodes are healthy, retry the cycle. Node pool cycling settings control surge and unavailability:

oci ce node-pool update --node-pool-id $NP --node-pool-cycling-details \
  '{"isNodeCyclingEnabled":true,"maximumSurge":"1","maximumUnavailable":"0"}'

maximumUnavailable: 0 with a surge of 1 adds a new node before removing an old one, which keeps capacity steady.

#Step 6: confirm

• All nodes Ready on the target version: kubectl get nodes -o custom-columns=NAME:.metadata.name,VER:.status.nodeInfo.kubeletVersion.
• No Pending pods; PDBs back to their normal values.
• Temporary node pool removed if you created one.

#Prevent it next time

• Policy: no PDB may have minAvailable equal to replicas; single-replica apps must not have a PDB that blocks drains.
• Upgrade non-production first and time the cycle.
• Keep 20% free IPs in worker and pod subnets.
• Use surge-based cycling (maximumUnavailable: 0).