AKS scheduling in depth: topology spread, taints, affinity and priority classes for resilient placement

Three replicas do not mean high availability if all three sit on one node. The Kubernetes scheduler places pods by default to fit resources, not to survive failures. This post shows how to tell it what you actually need.

#Start by seeing where pods are

kubectl get pods -n shop -l app=checkout -o wide
kubectl get nodes -L topology.kubernetes.io/zone,agentpool,kubernetes.azure.com/mode

If several replicas share a node or zone, keep reading.

#Topology spread constraints

The most useful tool. Spread across zones first, then nodes:

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector: { matchLabels: { app: checkout } }
    minDomains: 3
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector: { matchLabels: { app: checkout } }

• DoNotSchedule for zones: the service survives a zone outage, at the price of pending pods if a zone has no capacity.
• ScheduleAnyway for nodes: a preference, so a rollout is not blocked by a busy node.
• minDomains stops the scheduler treating "two zones available" as balanced when you need three.

Warning

Zone spread only works if the node pool spans zones (--zones 1 2 3) and the autoscaler can add nodes in each zone. A pool created without zones cannot be changed later; you need a new pool.

#Dedicated pools with taints and tolerations

Keep system pods, GPU jobs or noisy batch work on their own nodes.

az aks nodepool add -g rg-aks --cluster-name aks-prod -n batch \
  --node-taints workload=batch:NoSchedule --labels workload=batch \
  --enable-cluster-autoscaler --min-count 0 --max-count 20 --zones 1 2 3

spec:
  tolerations:
  - key: workload
    operator: Equal
    value: batch
    effect: NoSchedule
  nodeSelector:
    workload: batch

A toleration lets a pod onto tainted nodes; the node selector (or node affinity) makes it go there. You usually need both. For system pools, AKS supports CriticalAddonsOnly=true:NoSchedule so application pods stay off them.

#Affinity and anti-affinity

Use anti-affinity when two different workloads must not share a node (for example primary and replica of a database), or affinity to co-locate a cache with its app:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector: { matchLabels: { app: redis, role: primary } }
      topologyKey: kubernetes.io/hostname

Required inter-pod affinity rules are expensive for the scheduler in big clusters. Prefer topology spread for "spread my replicas" and keep affinity for relationships between different apps.

#Priority classes and preemption

When the cluster is full, priority decides who waits.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: business-critical }
value: 100000
preemptionPolicy: PreemptLowerPriority
globalDefault: false
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: batch-low }
value: 1000
preemptionPolicy: Never

Give checkout and payment pods priorityClassName: business-critical; batch jobs batch-low. Under pressure, low-priority pods are evicted to make room.

Caution

The scheduler tries to respect PodDisruptionBudgets when preempting, but only on a best-effort basis and may violate them. If a low-priority workload still matters, protect it with capacity, not just a PDB.

#Interaction with the cluster autoscaler

The autoscaler only adds nodes if a pending pod would fit on a new node of some pool. Check why pods are pending:

kubectl get events -A --field-selector reason=FailedScheduling --sort-by=.lastTimestamp | tail -20
kubectl describe pod <pending-pod> -n shop | sed -n '/Events/,$p'
kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

Common messages and meaning:

#Protect against voluntary disruption

Upgrades and node image updates drain nodes. Pair spread rules with a PDB:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: checkout-pdb, namespace: shop }
spec:
  minAvailable: 2
  selector: { matchLabels: { app: checkout } }

And set a max surge so upgrades add capacity before removing it:

az aks nodepool update -g rg-aks --cluster-name aks-prod -n apps --max-surge 33%

#Prevent it next time

• Bake topology spread and PDB into your Helm chart defaults.
• Use Azure Policy for AKS (Gatekeeper) to require PDBs and priority classes in production namespaces.
• Run a zone failure game day: cordon and drain all nodes in one zone and watch what stays up.