OKE pods stuck in ContainerCreating with VCN-native pod networking: pod subnet exhaustion, VNIC limits and max pods per node

#Symptoms

• Pods stay in ContainerCreating; kubectl describe pod shows events like failed to setup network for sandbox or IP allocation errors from the CNI.
• Pods stay Pending with Too many pods on every node, while CPU and memory are free.
• Scaling a deployment works up to a point, then stops; adding nodes helps briefly.

With VCN-native pod networking, each pod gets a real IP from a pod subnet in your VCN. Those IPs are secondary private IPs on extra VNICs attached to the worker node. Three things limit how many pods can run:

1. Free IPs in the pod subnet(s).
2. How many VNICs the node's shape can attach, and IPs per VNIC.
3. The node pool's max pods per node setting.

#Step 1: what does Kubernetes say?

kubectl get pods -A --field-selector=status.phase!=Running -o wide | head -30
kubectl describe pod <pod> -n <ns> | sed -n '/Events/,$p'

# Allocatable pods per node vs what's running
kubectl get nodes -o custom-columns=NODE:.metadata.name,PODS:.status.allocatable.pods
kubectl get pods -A -o wide --field-selector=status.phase=Running \
  | awk 'NR>1{print $8}' | sort | uniq -c

• Nodes at their allocatable pod count → max pods / VNIC capacity limit.
• Nodes below it but pods fail with network sandbox errors → usually pod subnet exhausted, or CNI errors.

#Step 2: check the pod subnet

POD_SUBNET=ocid1.subnet.oc1.eu-frankfurt-1.aaaa...
oci network subnet get --subnet-id $POD_SUBNET --query 'data.{cidr:"cidr-block",name:"display-name"}'

# Count used IPs
oci network private-ip list --subnet-id $POD_SUBNET --all --query 'length(data)'

Compare the count with the subnet size (a /24 has 256 addresses, minus the 3 OCI reserves). With VCN-native networking, OKE pre-allocates IPs on each node for its max pods, so a node uses IPs even for pods it isn't running yet. A /24 pod subnet runs out quickly with a few nodes set to 31+ pods each.

#Step 3: check VNIC capacity of the shape

Each node shape supports a limited number of VNICs, and each VNIC a limited number of secondary IPs (31 per VNIC). One VNIC is the node's primary. Flexible shapes scale VNIC count with OCPUs. Check the "VNIC limits" for your shape in the Compute shapes documentation.

oci compute instance list-vnics --instance-id <node-instance-ocid> \
  --query 'data[].{name:"display-name",subnet:"subnet-id",ip:"private-ip"}' --output table

If small shapes (1–2 OCPUs) can only attach one or two secondary VNICs, they can't host many pods no matter how big the subnet is.

#Step 4: check the node pool's pod settings

oci ce node-pool get --node-pool-id <node-pool-ocid> \
  --query 'data."node-config-details"."node-pool-pod-network-option-details"'

You'll see the CNI type, pod subnet IDs, pod NSGs and max-pods-per-node.

#Step 5: fix it

Add a CIDR and a bigger pod subnet:

oci network vcn add-vcn-cidr --vcn-id <vcn-ocid> --cidr-block 10.100.0.0/16

oci network subnet create --compartment-id <comp> --vcn-id <vcn-ocid> \
  --cidr-block 10.100.0.0/19 --display-name oke-pods-2 \
  --prohibit-public-ip-on-vnic true --route-table-id <pod-rt> --security-list-ids '["<pod-sl>"]'

Update the node pool (the JSON shape follows the API's NodePoolPodNetworkOptionDetails; check oci ce node-pool update --generate-param-json-input node-config-details for your CLI version):

oci ce node-pool update --node-pool-id <node-pool-ocid> --node-config-details '{
  "size": 6,
  "placementConfigs": [ ... existing placement configs ... ],
  "nodePoolPodNetworkOptionDetails": {
    "cniType": "OCI_VCN_IP_NATIVE",
    "podSubnetIds": ["<old-pod-subnet>", "<new-pod-subnet>"],
    "maxPodsPerNode": 31
  }
}'

Node pool settings apply to new nodes. Cycle existing nodes (scale up, cordon and drain old nodes, scale down, or use the node pool cycling feature) so they pick up the change.

Warning

Draining nodes moves running pods. Respect PodDisruptionBudgets and do it in a quiet period for stateful workloads.

#Step 6: don't forget pod security rules

New pod subnets need the same NSG/security list rules as the old ones: pod-to-pod, pod-to-node, pod-to-API-endpoint, and egress to the service gateway or NAT. Missing rules show up as pods that start but can't reach anything.

#Prevent it next time

• Size pod subnets for the maximum cluster size times max pods per node, plus headroom (a /19 or larger for production).
• Reserve a VCN CIDR range for future pod subnets.
• Set maxPodsPerNode from real usage, not the maximum.
• Alert on pod subnet usage (count of private IPs vs subnet size) from a scheduled script.