Home / Azure

AzureHow-toAdvanced

AKS networking deep dive: Azure CNI Overlay vs pod subnet, IP planning and egress at scale

How Azure CNI Overlay, flat Azure CNI with a dedicated pod subnet and kubenet differ in IP use, routing and limits; how to size CIDRs for growth; and how to plan egress so clusters do not hit SNAT or IP exhaustion later.

Deepak Bhardwaj4 min read

Applies toAzure Kubernetes Service with Azure CNI, Azure CNI Overlay, Azure CNI with dynamic pod IP allocation; Azure CLI 2.x (kubenet is being retired; check current AKS docs)

The network plugin is the AKS decision that is hardest to change later. Most re-platforming projects I have seen were caused by a VNet that ran out of IPs at 60% of the planned node count. This post is about choosing deliberately.

The options in one table

Azure CNI Overlay Azure CNI, pod subnet (dynamic IP allocation) Azure CNI, node subnet (legacy flat)
Pod IPs from Private overlay CIDR, not the VNet A dedicated VNet subnet The node subnet
VNet IPs consumed One per node One per pod + one per node One per pod + one per node, reserved up front by max-pods
Pods reachable directly from VNet / on-prem No (SNAT to node IP outbound) Yes Yes
Typical use Most new clusters Pods must be addressable from outside, per-pod NSG or firewall rules Existing clusters
Scale Large clusters with few VNet IPs Limited by subnet size Limited by subnet size and max-pods reservation

Inspect what an existing cluster uses:

az aks show -g rg-aks -n aks-prod --query "networkProfile.{plugin:networkPlugin, mode:networkPluginMode, podCidr:podCidr, svc:serviceCidr, policy:networkPolicy, dataplane:networkDataplane, outbound:outboundType}" -o table

Choose

  • Default to Overlay. It decouples pod count from VNet size. Pods reach Azure services and on-prem through node IPs, which works for almost every workload.
  • Pick pod subnet when something outside the cluster must connect to a pod IP directly, or security insists on firewall rules per pod range. Give each node pool its own pod subnet so you can apply different rules.
  • Avoid the legacy flat mode for new clusters; its up-front reservation (max-pods per node) wastes IPs.

Size the CIDRs

Overlay:

Node subnet: nodes_max + surge nodes during upgrade + internal LBs
   e.g. 300 nodes + 10% surge + 20 ILB IPs  -> /23 (510 usable) is comfortable
Pod CIDR (overlay): each node gets a /24 from it
   e.g. 10.244.0.0/16 -> 256 nodes; use a /13-/14 for larger clusters
Service CIDR: must not overlap VNet, peered VNets or on-prem; /16 is typical

Pod subnet:

Pod subnet = sum over pools of (max nodes x max pods per node), allocated in blocks
Node subnet = nodes + upgrade surge + ILBs

Warning

Upgrade surge needs extra node IPs. A subnet that is exactly full blocks upgrades with SubnetIsFull or similar errors at the worst moment. Leave at least 10-20% headroom.

The overlay pod CIDR and service CIDR must not overlap anything you route to, including peered hubs and on-prem ranges, or replies go to the wrong place.

Create

az aks create -g rg-aks -n aks-prod \
  --network-plugin azure --network-plugin-mode overlay \
  --pod-cidr 10.200.0.0/14 --service-cidr 10.0.0.0/16 --dns-service-ip 10.0.0.10 \
  --vnet-subnet-id $NODE_SUBNET_ID \
  --network-dataplane cilium \
  --outbound-type userDefinedRouting \
  --node-count 3 --max-pods 110

Pod subnet variant:

az aks nodepool add -g rg-aks --cluster-name aks-prod -n apps \
  --vnet-subnet-id $NODE_SUBNET_ID --pod-subnet-id $POD_SUBNET_APPS_ID --max-pods 50

Plan egress early

Outbound type Use when Watch
loadBalancer Simple, public egress allowed SNAT ports per node; set allocated outbound ports and IP count
managedNATGateway / userAssignedNATGateway Many outbound connections 64,512 SNAT ports per NAT IP; add IPs
userDefinedRouting Egress through Azure Firewall / NVA Firewall SNAT capacity, required AKS FQDN rules

Check SNAT pressure on the load balancer:

AzureMetrics
| where ResourceProvider == "MICROSOFT.NETWORK" and MetricName in ("UsedSnatPorts","SnatConnectionCount")
| summarize avg(Average), max(Maximum) by MetricName, bin(TimeGenerated, 5m)

Migration paths

  • Kubenet and legacy Azure CNI clusters can be upgraded in place to Overlay on supported versions:
az aks update -g rg-aks -n aks-old --network-plugin azure --network-plugin-mode overlay --pod-cidr 10.200.0.0/14

This reimages nodes. Test in non-prod and check the current list of prerequisites (no Windows node pool restrictions, policy engine, etc.).

  • Switching between pod subnet and Overlay generally means a new cluster and a blue/green cutover.

Prevent it next time

  • Write the IP plan down: node subnet, pod CIDR, service CIDR, growth target, surge. Get the network team to reserve it.
  • Alert on subnet usage and on node count approaching the plan.
  • Choose the network policy engine (Cilium dataplane or Azure NPM/Calico) at creation and test NetworkPolicies in staging.

Related fixes