Home / AWS

AWSTroubleshootingIntermediate

Sev1: Auto Scaling group stuck launching and terminating instances in a loop

Stop an EC2 Auto Scaling group that keeps replacing instances. Read scaling activities, find whether ELB health checks, a short grace period, a bad launch template or user data is the cause, freeze replacements and roll back.

Deepak Bhardwaj2 min read

Applies toAmazon EC2 Auto Scaling, launch templates, Application Load Balancer target groups, AWS CLI v2

Symptoms

  • Instances go InService, fail health checks, terminate, and new ones launch — over and over.
  • Capacity never stabilises; the load balancer has few or no healthy targets.
  • It often starts right after a new AMI, launch template version or instance refresh.

1. Read the scaling activities

aws autoscaling describe-scaling-activities --auto-scaling-group-name asg-web --max-items 15 \
  --query "Activities[].{time:StartTime, status:StatusCode, cause:Cause, desc:Description}" --output table
Cause text mentions Meaning
ELB system health check failure Target group health check fails
EC2 health check ... impaired Instance status check fails (boot, kernel, disk)
Launching a new EC2 instance. Status Reason: ... with an error Launch fails (quota, capacity, AMI, IAM, subnet IPs)
instance refresh A refresh is replacing instances with a bad version

2. Freeze replacements while you investigate

aws autoscaling suspend-processes --auto-scaling-group-name asg-web \
  --scaling-processes ReplaceUnhealthy HealthCheck AZRebalance

Healthy-or-not instances now stay, so you can log in and look. Remember to resume later.

3. Health check failures

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names asg-web \
  --query "AutoScalingGroups[0].{type:HealthCheckType, grace:HealthCheckGracePeriod, lt:LaunchTemplate}"
aws elbv2 describe-target-health --target-group-arn <tg-arn> \
  --query "TargetHealthDescriptions[].{id:Target.Id, state:TargetHealth.State, reason:TargetHealth.Reason, desc:TargetHealth.Description}" --output table
  • Grace period too short: the app needs longer to start than HealthCheckGracePeriod. Raise it (for example 300s) or use lifecycle hooks / warm pools.
  • Health check path wrong after a release: Target.ResponseCodeMismatch or Target.Timeout.
  • Security group: instances must accept the health check port from the load balancer's security group.

On an instance, check user data and app logs:

sudo cat /var/log/cloud-init-output.log | tail -50
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/health

4. Roll back the change

Point the ASG at the last good launch template version:

aws ec2 describe-launch-template-versions --launch-template-id lt-0abc --query "LaunchTemplateVersions[].[VersionNumber,CreateTime,VersionDescription]" --output table
aws autoscaling update-auto-scaling-group --auto-scaling-group-name asg-web \
  --launch-template LaunchTemplateId=lt-0abc,Version=12

If an instance refresh is in progress:

aws autoscaling cancel-instance-refresh --auto-scaling-group-name asg-web
aws autoscaling rollback-instance-refresh --auto-scaling-group-name asg-web

Then start a new refresh to the good version if needed.

5. Resume

aws autoscaling resume-processes --auto-scaling-group-name asg-web

Warning

Resume only after targets are healthy. Resuming ReplaceUnhealthy with sick instances restarts the loop.

Prevent it next time

  • Instance refresh with checkpoints and auto rollback and alarms tied to it.
  • Grace period based on measured start-up time; readiness endpoints that return 200 only when the app is ready.
  • Test new AMIs in a canary ASG before production.

Related fixes