Rolling out a new AMI with EC2 Auto Scaling instance refresh: checkpoints, auto rollback and failed refreshes

Goal: move an Auto Scaling group to a new AMI or launch template version without an outage, and be able to back out if the new build is bad.

Updating the launch template doesn't change running instances. Instance refresh replaces them in batches, and the group's own configuration only changes when the refresh succeeds.

Step 1: pre-flight checks

Set the group name and check its current state:

ASG=web-asg

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names "$ASG" \
  --query 'AutoScalingGroups[0].{LT:LaunchTemplate,MIP:MixedInstancesPolicy.LaunchTemplate.LaunchTemplateSpecification,Min:MinSize,Max:MaxSize,Desired:DesiredCapacity,HealthCheck:HealthCheckType,Grace:HealthCheckGracePeriod,Warmup:DefaultInstanceWarmup,Suspended:SuspendedProcesses[].ProcessName}'

Check	What you want	Why
`Version` on the launch template	A number, not `$Latest` or `$Default`	Rollback isn't supported otherwise
`ImageId` in the template	An AMI ID, not an SSM parameter	Rollback isn't supported otherwise
`HealthCheck`	`ELB` for groups behind a load balancer	With `EC2`, an instance that fails its target group checks can still count as healthy
`Suspended`	No `Launch` or `Terminate`	The refresh can't replace anything if these are suspended
`Desired`	More than 1	A single-instance group can have an outage

Pin the group to a numbered version if needed. This doesn't replace any instances:

aws autoscaling update-auto-scaling-group --auto-scaling-group-name "$ASG" \
  --launch-template LaunchTemplateName=web-lt,Version=7

Find instances that can block the refresh:

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names "$ASG" \
  --query "AutoScalingGroups[0].Instances[?ProtectedFromScaleIn==\`true\` || LifecycleState=='Standby'].[InstanceId,LifecycleState,ProtectedFromScaleIn]" \
  --output table

Check that no refresh is already running, because a group can only have one active refresh:

aws autoscaling describe-instance-refreshes --auto-scaling-group-name "$ASG" \
  --max-records 5 --query 'InstanceRefreshes[].[InstanceRefreshId,Status,StartTime]' --output table

Warning
Scale-in protected instances are often doing long-running work, and Standby instances are often being debugged. Find out why they're in that state before you use Refresh or Terminate. Those settings replace or terminate them.

Step 2: create the new launch template version

NEW_VERSION=$(aws ec2 create-launch-template-version \
  --launch-template-name web-lt --source-version 7 \
  --version-description "AMI build 2026-11-04" \
  --launch-template-data '{"ImageId":"ami-0123456789abcdef0"}' \
  --query 'LaunchTemplateVersion.VersionNumber' --output text)
echo "$NEW_VERSION"

Launch one instance from this version and test it before you touch the group.

Step 3: choose the preferences

Save this as refresh.json and set Version to the value of NEW_VERSION:

{
  "AutoScalingGroupName": "web-asg",
  "DesiredConfiguration": {
    "LaunchTemplate": { "LaunchTemplateName": "web-lt", "Version": "8" }
  },
  "Preferences": {
    "MinHealthyPercentage": 100,
    "MaxHealthyPercentage": 120,
    "InstanceWarmup": 180,
    "SkipMatching": true,
    "CheckpointPercentages": [20, 50, 100],
    "CheckpointDelay": 900,
    "BakeTime": 1800,
    "AutoRollback": true,
    "AlarmSpecification": { "Alarms": ["web-5xx-high"] },
    "ScaleInProtectedInstances": "Wait",
    "StandbyInstances": "Wait"
  }
}

Preference	Default	Notes
`MinHealthyPercentage`	90, or the group's instance maintenance policy	0-100. Higher is safer but slower
`MaxHealthyPercentage`	100, or the group's instance maintenance policy	100-200. Above 100 launches before terminating. Max minus min can't exceed 100
`InstanceWarmup`	`DefaultInstanceWarmup`, else the health check grace period	Time from `InService` until the instance finishes initializing. The refresh waits this long before moving on
`SkipMatching`	`false` in the CLI and SDK, enabled in the console	Skips instances that already match, which is useful for a re-run
`CheckpointPercentages`	None	Ascending. End with 100 to replace every instance
`CheckpointDelay`	3600 seconds if checkpoints are set	0-172800 seconds
`BakeTime`	0	Seconds to wait after the last replacement. The status is `Baking`
`AutoRollback`	`false`	Rolls back on failure
`AlarmSpecification`	None	Up to 10 alarms
`ScaleInProtectedInstances`, `StandbyInstances`	`Wait` in the CLI and SDK, `Ignore` in the console	`Wait` waits one hour, then fails the refresh

A few points catch people out:

A checkpoint is a timed pause, not an approval gate. After CheckpointDelay the refresh continues on its own. To stop it, roll back or cancel.
An explicit MaxHealthyPercentage of 100 isn't the same as leaving it out. If you set it and both bounds can't be honoured, Auto Scaling launches first and temporarily exceeds desired capacity. If you leave it out, it terminates and launches at the same time.
An alarm must be in OK when you start. ALARM or INSUFFICIENT_DATA causes a start error.
Launching first needs headroom. Check instance quotas and free IP addresses in the subnets.
Mixed instances policies are different. Set DesiredConfiguration.MixedInstancesPolicy instead of LaunchTemplate, and specify the whole policy. Check the API reference for your group's shape.

Caution
Don't run a refresh on a single-instance group with the console's default settings. AWS warns it can cause an outage. Use MinHealthyPercentage 100 with MaxHealthyPercentage above 100 so the new instance launches first.

Step 4: start and watch

REFRESH_ID=$(aws autoscaling start-instance-refresh \
  --cli-input-json file://refresh.json --query InstanceRefreshId --output text)

aws autoscaling describe-instance-refreshes --auto-scaling-group-name "$ASG" \
  --instance-refresh-ids "$REFRESH_ID" \
  --query 'InstanceRefreshes[0].{Status:Status,Pct:PercentageComplete,Left:InstancesToUpdate,Reason:StatusReason}'

The status moves through Pending, InProgress, Baking if you set a bake time, then Successful. An instance counts towards PercentageComplete once it's healthy and its warmup has passed.

Check which instances run which version:

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names "$ASG" \
  --query 'AutoScalingGroups[0].Instances[].[InstanceId,LaunchTemplate.Version,LifecycleState,HealthStatus,AvailabilityZone]' \
  --output table

Use each checkpoint to look at error rates, latency and application logs on the new instances.

Step 5: back out if the build is bad

Situation	Command	Result
Still running, new build is bad	`rollback-instance-refresh`	Replaces deployed instances with the configuration last saved on the group
Still running, you want to stop but keep what's replaced	`cancel-instance-refresh`	Stops the refresh, makes no rollback. The group stays on mixed versions
Already `Successful`	New `start-instance-refresh` with the old version	Rollback is no longer possible

aws autoscaling rollback-instance-refresh --auto-scaling-group-name "$ASG"

aws autoscaling cancel-instance-refresh --auto-scaling-group-name "$ASG"

If the rollback returns IrreversibleInstanceRefresh, the refresh had no desired configuration, the template uses an SSM parameter for its AMI, or the group uses $Latest or $Default. Start a new refresh with the last good numbered version instead.

Warning
Rollback goes to the configuration saved before the refresh. If that configuration can't launch, for example because its AMI has been deregistered, the rollback eventually fails and the group may be unable to launch instances. Keep the previous AMI and its snapshots until the new build has proved itself.

When a refresh stalls or fails

Start with StatusReason, then the scaling activities:

aws autoscaling describe-scaling-activities --auto-scaling-group-name "$ASG" \
  --max-items 10 --query 'Activities[].[StartTime,StatusCode,StatusMessage]' --output table

Symptom	Cause	Fix
`InstanceRefreshInProgress` on start	A refresh is already active	Wait, or cancel the active one
Start error mentioning an alarm	Alarm isn't in `OK`	Fix the alarm, or wait until it has data
`Failed` after about an hour, protected or Standby instances remain	`Wait` timed out	Remove protection, exit Standby, or use `Ignore`
New instances launch then get replaced	Failing ELB health checks, grace period too short or a bad build	Check the app and target health. Raise the grace period if the app just starts slowly
No new instances launch	Launch failures such as AMI, capacity, quota or KMS	Read `StatusMessage` in the scaling activities
Instances sit in `Pending:Wait` or `Terminating:Wait`	Lifecycle hook isn't completing	Fix the hook's handler, or check its timeout
Very slow progress	Long warmup, high minimum healthy, large group	Lower `InstanceWarmup` if it's longer than needed. Raise `MaxHealthyPercentage`
`Successful` at once with nothing replaced	`SkipMatching` and instances already match	Check that `DesiredConfiguration` has the new version
`RollbackFailed`	Saved configuration can't launch	Fix the launch failure, then start a new refresh with a known good version

With auto rollback on, a refresh keeps trying to finish failing replacements for an hour before it fails and rolls back. For a clearly bad build, don't wait. Run rollback-instance-refresh yourself.

Prevent it next time

Pin groups to numbered launch template versions and use AMI IDs if you want rollback.
Use HealthCheckType ELB and an application health check, so bad builds fail the refresh instead of passing.
Set DefaultInstanceWarmup on the group based on measured start time:

aws autoscaling update-auto-scaling-group --auto-scaling-group-name "$ASG" \
  --default-instance-warmup 180

Attach a CloudWatch alarm on a customer-facing metric, not only CPU.
Send refresh events to your on-call channel with an EventBridge rule. Add your SNS topic or other target with put-targets:

aws events put-rule --name asg-instance-refresh-events \
  --event-pattern '{"source":["aws.autoscaling"],"detail-type":[{"prefix":"EC2 Auto Scaling Instance Refresh"}]}'

Refresh non-production first with the same refresh.json, so the settings get tested before production.

Rolling out a new AMI with EC2 Auto Scaling instance refresh: checkpoints, auto rollback and failed refreshes

#Step 1: pre-flight checks

#Step 2: create the new launch template version

#Step 3: choose the preferences

#Step 4: start and watch

#Step 5: back out if the build is bad

#When a refresh stalls or fails

#Prevent it next time

EC2 Spot interruptions: catch the two-minute notice, drain cleanly and keep capacity

EC2 instance won't launch: capacity, quota and RunInstances errors decoded

Patching EC2 fleets with SSM Patch Manager

Step 1: pre-flight checks

Step 2: create the new launch template version

Step 3: choose the preferences

Step 4: start and watch

Step 5: back out if the build is bad

When a refresh stalls or fails

Prevent it next time