Static stability in one sentence
The system keeps working through an AZ failure without needing to make changes (launch instances, update DNS, call APIs). Changes may fail or be slow during a large event.
1. Capacity: N+1 across zones
| Design | Behaviour when one of three AZs fails |
|---|---|
| 3 AZs, each sized for 33% of peak | Remaining capacity 67%: overload until Auto Scaling adds instances (if it can) |
| 3 AZs, each sized for 50% of peak | Remaining capacity 100%: no scaling needed |
| 2 AZs, each sized for 100% | Survives, at double cost |
Over-provisioning by one zone's share is the price of static stability. Decide per tier (Tier 1 yes; internal tools may accept degraded service).
2. Data layer
| Service | Make sure |
|---|---|
| RDS | Multi-AZ; app reconnect logic and short DNS TTL caching |
| Aurora | Readers in other AZs; failover priority tiers set |
| ElastiCache/MemoryDB | Multi-AZ with replicas, automatic failover |
| EFS | Regional (not One Zone) for production |
| EBS-based self-managed databases | Replication to another AZ (EBS is single-AZ) |
| Kafka/MSK, OpenSearch | Replication factor and zone awareness configured |
3. Avoid hidden single-AZ dependencies
- One NAT Gateway for all AZs (common!). Use one per AZ with same-AZ routing.
- Singleton components: a cron instance, a license server, a bastion, a self-managed DNS forwarder.
- EKS: CoreDNS replicas and critical add-ons spread across zones; PDBs and topology spread constraints.
- Third-party appliances (firewalls) in one AZ.
# NAT per AZ check
aws ec2 describe-nat-gateways --filter Name=state,Values=available \
--query 'NatGateways[].[NatGatewayId,SubnetId]' --output table
4. Detect partial failures
An impaired AZ often looks "up" to simple health checks but has higher latency or errors. Detect per zone:
- ALB metrics per AZ (
HTTPCode_Target_5XX_Count,TargetResponseTimewith AvailabilityZone dimension). - Deep health checks that test dependencies carefully (avoid all instances failing health checks because a shared dependency hiccups, which causes mass removal).
- Canaries (CloudWatch Synthetics) per zone where feasible.
5. Move traffic away: zonal shift
Application Recovery Controller zonal shift moves load balancer traffic away from an AZ for a set duration, with no changes to your infrastructure:
aws arc-zonal-shift start-zonal-shift --resource-identifier <alb-or-nlb-arn> \
--away-from euw1-az2 --expires-in 2h --comment "Elevated 5xx in euw1-az2"
Zonal autoshift lets AWS shift traffic automatically when it detects an AZ impairment, and runs practice shifts. Requires you to have capacity in the other zones (static stability again).
Cross-zone load balancing and zonal shift interplay: check the documented behaviour for your load balancer type.
6. Practise
- Game day: shift traffic away from one AZ in production during low traffic, watch latency and errors.
- AWS Fault Injection Service scenarios for AZ power interruption (with stop conditions).
- Record results and fix what broke.
Design review questions
- If an AZ disappears now, what breaks without anyone doing anything?
- Which recovery steps need control-plane APIs (launch, scale, DNS change)?
- What singleton components exist, and in which AZ?
- When did you last move production traffic away from an AZ?
Tip
Static stability applies to people too: runbooks shouldn't require consoles or tools hosted in the impaired zone or region. Keep break-glass access and runbooks reachable from elsewhere.
Checklist
- Capacity sized N+1 across AZs for Tier 1
- Data services configured Multi-AZ with app reconnect logic
- Single-AZ dependencies (NAT, singletons, appliances) removed
- Per-AZ health signals
- Zonal shift / autoshift configured
- Game day or FIS experiment done