Home / AWS

AWSChecklistIntermediate

Highly available workloads on EKS across three AZs

Node, pod, storage and load balancer settings that keep EKS applications serving through node failures, upgrades and an AZ outage, including EBS volume zoning and Karpenter disruption controls.

Deepak Bhardwaj2 min read

Applies toAmazon EKS with managed node groups or Karpenter, AWS Load Balancer Controller

Nodes

  • Node groups (or Karpenter NodePools) spanning three AZs.
  • With managed node groups and EBS volumes: one node group per AZ, so the cluster autoscaler can add a node in the right AZ for a pending pod with a zonal volume.
  • Capacity headroom or fast scaling to absorb the loss of an AZ.
  • Karpenter: disruption budgets on NodePools so consolidation doesn't drain too many nodes at once.
apiVersion: karpenter.sh/v1
kind: NodePool
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    budgets:
      - nodes: "10%"

Pods

  • 3+ replicas with topologySpreadConstraints on topology.kubernetes.io/zone.
  • PodDisruptionBudgets (maxUnavailable: 1 is a safe default).
  • Readiness, liveness and startup probes.
  • preStop hook (e.g. sleep 10) and terminationGracePeriodSeconds long enough to finish in-flight requests, because target deregistration takes a moment.

Load balancing

  • AWS Load Balancer Controller with IP targets (alb.ingress.kubernetes.io/target-type: ip) so the ALB sends traffic straight to pods.
  • Pod readiness gates (namespace label elbv2.k8s.aws/pod-readiness-gate-inject=enabled) so rolling updates wait until new pods are healthy in the target group.
  • ALB subnets in all three AZs.

Storage

  • EBS volumes are AZ-bound. A StatefulSet pod with an EBS volume can't move to another AZ. Use volumeBindingMode: WaitForFirstConsumer.
  • For data that must survive an AZ outage: app-level replication (a database with replicas in each AZ), EFS (regional, multi-AZ), or managed services (RDS/Aurora, ElastiCache) outside the cluster.

Cluster services

  • CoreDNS with 2+ replicas spread across AZs (and NodeLocal DNSCache for heavy DNS use).
  • Critical add-ons on a dedicated system node group.
  • Control plane: managed by AWS across AZs; keep supported versions to stay on standard support.

Test

kubectl get nodes -L topology.kubernetes.io/zone
kubectl get pods -o wide -l app=orders-api

Cordon and drain all nodes in one AZ during a test window, or use AWS Fault Injection Service with EKS actions, and check that error rates stay flat.

Tip

Watch cross-AZ data transfer costs. Topology-aware routing (service.kubernetes.io/topology-mode: Auto) keeps service traffic in the same AZ when enough endpoints exist there.

Related fixes