Home / OCI

OCITroubleshootingAdvanced

OKE: nodes NotReady and pods stuck Pending

Troubleshoot Oracle Container Engine for Kubernetes worker nodes that don't become Ready and pods that won't schedule, from security rules and capacity to pod networking IPs.

Deepak Bhardwaj2 min read

Applies toOCI Kubernetes Engine (OKE) enhanced/basic clusters, flannel or VCN-native pod networking

Get access

oci ce cluster create-kubeconfig --cluster-id $CLUSTER --file $HOME/.kube/config \
  --region ap-mumbai-1 --token-version 2.0.0 --kube-endpoint PRIVATE_ENDPOINT
kubectl get nodes -o wide

Nodes never appear, or stay NotReady

1. Node pool and compute status

oci ce node-pool get --node-pool-id $NP --query 'data.nodes[].{name:name,state:"lifecycle-state",details:"lifecycle-details",err:"node-error"}'

A node-error such as Out of host capacity means OCI couldn't place the instance. Try another fault domain, AD or shape (see the capacity post).

2. Security rules between workers and the control plane

Worker nodes must reach the Kubernetes API endpoint subnet on TCP 6443 and TCP 12250, and the control plane must reach the workers. If you created custom security lists or NSGs, compare them with the OKE network resource configuration in the documentation. Missing rules are the top cause of NotReady nodes on custom VCNs.

3. Outbound access for nodes

Private worker subnets need:

  • A NAT gateway (to pull public images), and/or
  • A service gateway to "All <region> Services in Oracle Services Network" (for OCIR and Object Storage).

4. On the node

# through the Bastion service, or a console connection
sudo journalctl -u kubelet --no-pager | tail -50
sudo crictl ps -a | head

Pods stuck Pending

kubectl describe pod <pod> -n <ns> | sed -n '/Events/,$p'
Event Cause Fix
0/3 nodes are available: insufficient cpu/memory Requests exceed free capacity Scale the node pool, or turn on the cluster autoscaler add-on
node(s) had untolerated taint Taints on nodes Add tolerations, or use the right node pool
pod has unbound immediate PersistentVolumeClaims Block volume PVC can't be provisioned Check the storage class, and that the PVC and node are in the same AD
failed to assign an IP address (VCN-native networking) Pod subnet is out of IPs Larger pod subnet, or fewer maxPodsPerNode
oci ce node-pool update --node-pool-id $NP --size 5

Block volumes and ADs

OCI block volumes are AD-local. A pod with a PVC in AD-1 can't run on a node in AD-2. Use WaitForFirstConsumer binding in the storage class, or keep stateful workloads in node pools pinned to one AD.

Tip

When you create a new cluster, use the console's Quick Create network as a reference. Compare its security lists with your custom VCN, rule by rule, when nodes won't join.