OKE Cluster Autoscaler not adding or removing nodes: a diagnostic checklist

#Symptom

One of two things:

• No scale-up: pods are Pending with FailedScheduling (Insufficient cpu, Insufficient memory), and the node pool size doesn't change.
• No scale-down: nodes run at low utilization for hours and the pool stays at its peak size.

The Cluster Autoscaler (CA) only works on managed node pools. It doesn't manage virtual node pools, which scale per pod anyway. Work through the checklist in order; most failures are in the first four sections.

export CLUSTER=ocid1.cluster.oc1..aaaa
kubectl get pods -A --field-selector=status.phase=Pending
kubectl -n kube-system get deploy,pods -l app=cluster-autoscaler -o wide

Label selectors and deployment names vary between the add-on and self-managed installs; kubectl -n kube-system get deploy | grep -i autoscaler finds it either way.

#1. Is the autoscaler running, and only once?

• The add-on is installed and ACTIVE, or a self-managed deployment exists, but not both. Two autoscalers fight over pool sizes.

oci ce cluster list-addons --cluster-id $CLUSTER \
  --query 'data[].{name:name,state:"lifecycle-state",version:"current-installed-version"}' --output table
oci ce cluster get-addon --cluster-id $CLUSTER --addon-name ClusterAutoscaler
kubectl -n kube-system get deploy | grep -i autoscaler

• The CA version matches the cluster's Kubernetes minor version. A mismatched self-managed image is a common cause after a control plane upgrade.
• Pods aren't crash-looping. Check the first 50 log lines for auth and config errors:

kubectl -n kube-system logs deploy/cluster-autoscaler --tail=200

• With multiple replicas, find the leader (only it acts). Read the logs from that pod.

#2. Does the CA know about the node pool?

• The nodes configuration lists every pool you expect to scale, in min:max:nodepool-ocid format, comma-separated:

nodes: "2:10:ocid1.nodepool.oc1..aaaa,0:20:ocid1.nodepool.oc1..bbbb"

• Or nodeGroupAutoDiscovery is set instead (compartment and node pool tags), on a CA version that supports it. Never set both.
• The pool's current size is below max. A pool already at max produces "max node group size reached" in the logs.
• The node pool OCID is current. Re-created pools get new OCIDs, and stale ones fail silently in some versions.

oci ce node-pool get --node-pool-id $NP \
  --query 'data.{size:"node-config-details".size,state:"lifecycle-state",k8s:"kubernetes-version"}'

Update add-on settings with oci ce cluster update-addon --cluster-id $CLUSTER --addon-name ClusterAutoscaler --configurations file://ca-config.json, where the file is a list of {"key": ..., "value": ...} pairs.

#3. Can it call the OCI API?

The CA resizes node pools through the Container Engine API. The authType setting decides which identity it uses:

• Logs show no 401, 404 NotAuthorizedOrNotFound or "failed to get node pool" errors.
• For instance: the dynamic group matches the worker nodes (for example by compartment), and policies grant the dynamic group access in the node pool compartment.
• For workload: policies use the request principal conditions:

Allow any-user to manage cluster-node-pools in compartment <compartment> where ALL {request.principal.type='workload', request.principal.namespace='kube-system', request.principal.service_account='cluster-autoscaler', request.principal.cluster_id='<cluster-ocid>'}
Allow any-user to manage instance-family in compartment <compartment> where ALL {request.principal.type='workload', request.principal.namespace='kube-system', request.principal.service_account='cluster-autoscaler', request.principal.cluster_id='<cluster-ocid>'}
Allow any-user to use subnets in compartment <compartment> where ALL {request.principal.type='workload', request.principal.namespace='kube-system', request.principal.service_account='cluster-autoscaler', request.principal.cluster_id='<cluster-ocid>'}
Allow any-user to read virtual-network-family in compartment <compartment> where ALL {request.principal.type='workload', request.principal.namespace='kube-system', request.principal.service_account='cluster-autoscaler', request.principal.cluster_id='<cluster-ocid>'}
Allow any-user to use vnics in compartment <compartment> where ALL {request.principal.type='workload', request.principal.namespace='kube-system', request.principal.service_account='cluster-autoscaler', request.principal.cluster_id='<cluster-ocid>'}
Allow any-user to inspect compartments in compartment <compartment> where ALL {request.principal.type='workload', request.principal.namespace='kube-system', request.principal.service_account='cluster-autoscaler', request.principal.cluster_id='<cluster-ocid>'}

• Network and node subnets in another compartment are covered by policies in that compartment too.
• Worker nodes can reach OCI APIs (service gateway or NAT route from the node subnet). A private cluster with no route to the Container Engine endpoint has every API call time out.

#4. Would a new node actually fit the pod?

The CA simulates scheduling on a template node from the pool. If the simulated node can't run the pod, it doesn't scale, and the pod events say so.

kubectl describe pod <pending-pod> | sed -n '/Events/,$p'
# look for: "pod didn't trigger scale-up" and the reason per node group
kubectl -n kube-system get configmap cluster-autoscaler-status -o yaml

• The pod's requests fit the pool's shape after system reservations and DaemonSets. A pod requesting 14 GiB on a 16 GB node often doesn't fit.
• nodeSelector, node affinity and tolerations match the pool's labels and taints (node pool initial-node-labels and taint settings), not just the labels of existing nodes.
• Architecture matches: arm64-only images on an Ampere pool, amd64 on x86.
• PVC availability domain: a pod bound to a block volume in AD-1 can only use a pool with placement in AD-1.
• Topology spread constraints and pod anti-affinity can be met by adding a node in a placement the pool actually has.

#5. Did the scale-up start but the node never joined?

• Check node pool work requests for launch failures:

oci ce work-request list -c $C --resource-id $NP --all \
  --query 'data[].{op:"operation-type",status:status,started:"time-started"}' --output table
oci ce work-request-error list --work-request-id $WR -c $C

• Look for out-of-host-capacity, LimitExceeded or compartment quota errors. Add placement in more ADs, use a capacity reservation or add a second pool with a different shape.
• Subnet IP exhaustion: node subnet, and with VCN-native pod networking, the pod subnet too.
• Nodes that launch but stay NotReady beyond maxNodeProvisionTime (default 15 minutes) are treated as failed, and the CA backs off that pool. Fix the boot or join problem first (cloud-init, image, NSG to the API endpoint).

#6. Why won't it scale down?

kubectl -n kube-system logs deploy/cluster-autoscaler --tail=500 | grep -iE "scale.?down|unremovable|not be removed"
kubectl get pdb -A

Warning

Lowering scaleDownUnneededTime and utilization thresholds aggressively on clusters with stateful or slow-starting workloads causes eviction churn. Change one parameter at a time and watch restarts and p99 latency.

#7. Don't let OKE operations fight the CA

• Don't change node pool size manually (Console, Terraform size) on CA-managed pools. In Terraform, ignore changes to the size attribute.
• During node pool cycling or upgrades, expect temporary over-provisioning. Leave headroom in max for surge nodes.
• Use balanceSimilarNodeGroups when you split one logical pool across several pools (for example one per AD).

#Prevent it next time

• Manage the add-on configuration in code, with the nodes list generated from the pool OCIDs.
• Alert on pods Pending for more than 10 minutes and on CA log errors, not just on node count.
• After every Kubernetes upgrade, confirm the CA version and run a scale-up test (a deployment with large requests) in a non-production cluster.
• Review PDBs and safe-to-evict annotations quarterly; they are the usual reason clusters stop shrinking.