Home / AWS

AWSRunbookAdvanced

Sev1: every ALB target is unhealthy and users get 503. Health check triage in ten minutes

The Application Load Balancer returns 503 because no target passes health checks. Read the health reason codes, separate a broken app from a broken health check or network change, restore traffic, and harden checks so one bad dependency cannot take the site down.

Deepak Bhardwaj4 min read

Applies toElastic Load Balancing Application Load Balancer, EC2 / ECS / EKS (AWS Load Balancer Controller) targets, AWS CLI v2

Symptoms

  • Clients receive 503 Service Temporarily Unavailable from the ALB (not from the app).
  • HealthyHostCount is 0 for the target group; UnHealthyHostCount equals the number of targets.
  • Often right after a deploy, a security group change, a certificate change on the targets, or a dependency outage.

Step 1: read the reason codes (2 minutes)

aws elbv2 describe-target-health --target-group-arn $TG \
  --query "TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State,TargetHealth.Reason,TargetHealth.Description]" --output table
Reason Meaning Look at
Target.Timeout No response within the timeout Security groups, NACLs, app hung or not listening
Target.ResponseCodeMismatch App answered with a code not in the matcher (e.g. 302, 401, 500) Health path, auth on the path, app errors
Target.FailedHealthChecks Generic failure, often connection refused Process down, wrong port
Elb.InitialHealthChecking New targets still checking Wait; check interval and threshold
Target.NotRegistered / Target.NotInUse Not registered, or AZ not enabled on the ALB Registration, ALB subnets

Step 2: test from inside the VPC

From a host in the same subnet as the ALB (or a target itself):

curl -sv -o /dev/null -w "%{http_code} %{time_total}\n" http://10.0.12.34:8080/health

Check the health check settings match what works:

aws elbv2 describe-target-groups --target-group-arns $TG \
  --query "TargetGroups[].{Proto:HealthCheckProtocol,Port:HealthCheckPort,Path:HealthCheckPath,Matcher:Matcher.HttpCode,Timeout:HealthCheckTimeoutSeconds,Interval:HealthCheckIntervalSeconds,Unhealthy:UnhealthyThresholdCount}"

Then match the reason code with the cause:

  • Timeout, but curl from a target works: target SG does not allow the ALB SG on the health port. Confirm:
aws ec2 describe-security-groups --group-ids sg-app \
  --query "SecurityGroups[].IpPermissions[].{Port:FromPort,From:UserIdGroupPairs[].GroupId}"
  • ResponseCodeMismatch 302/401: the health path now redirects to login after a deploy. Point the check at an unauthenticated endpoint.
  • ResponseCodeMismatch 500/503: the app health endpoint checks a dependency (database, cache) that is down. Every target fails together.
  • FailedHealthChecks: process crashed or listens on a different port after a config change.

Step 3: restore traffic

Pick the fastest safe action for the cause:

Cause Action
Bad deploy Roll back (aws deploy stop-deployment --deployment-id <id> --auto-rollback-enabled, ECS previous task definition, kubectl rollout undo)
SG change Re-add the ALB SG to the target SG inbound on the traffic and health ports
Health path changed aws elbv2 modify-target-group --target-group-arn $TG --health-check-path /healthz --matcher HttpCode=200
Dependency down, app could serve partially Temporarily point the check at a shallow liveness path 
aws ec2 authorize-security-group-ingress --group-id sg-app --protocol tcp --port 8080 --source-group sg-alb

Tip

When all targets in a target group are unhealthy, the ALB "fails open" and routes to all of them anyway. If you still see 503s, the targets are genuinely not answering, or there are no registered targets in the enabled AZs. Do not count on fail-open as a safety net.

Step 4: check the edges

  • aws elbv2 describe-load-balancers --names alb-web --query "LoadBalancers[].AvailabilityZones[].ZoneName": the ALB must have subnets in the AZs where targets run.
  • For ECS: aws ecs describe-services events show "unhealthy in target group" and tasks being replaced in a loop. Raise healthCheckGracePeriodSeconds if the app is slow to start.
  • For EKS: the AWS Load Balancer Controller logs show registration errors; check readiness gates and the target type (instance vs ip).

Step 5: confirm

aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HealthyHostCount \
  --dimensions Name=TargetGroup,Value=targetgroup/tg-web/0123456789abcdef Name=LoadBalancer,Value=app/alb-web/0123456789abcdef \
  --statistics Minimum --period 60 --start-time $(date -u -d '-30 min' +%FT%TZ) --end-time $(date -u +%FT%TZ)
  • HealthyHostCount back to normal; HTTPCode_ELB_5XX_Count at zero; real user requests succeed.

Prevent it next time

  • Split health endpoints: liveness/shallow for the load balancer (process up, can serve), deep checks for monitoring only. A database outage should degrade the site, not remove every target.
  • Alarm on HealthyHostCount < N per target group and HTTPCode_ELB_5XX_Count.
  • Manage SGs as code with the ALB-to-target rule in the same module, so it cannot be removed separately.
  • Deploys with health-based automatic rollback (CodeDeploy, ECS deployment circuit breaker).

Related fixes