EC2 Spot interruptions: catch the two-minute notice, drain cleanly and keep capacity

Spot Instances save money, but EC2 can take them back. You get a two-minute interruption notice, and sometimes an earlier rebalance recommendation. If nothing on the instance reacts, in-flight requests fail, jobs restart from zero and the Auto Scaling group sits below capacity until a replacement boots.

This post covers four things: confirming that interruptions are the cause, catching the signals, draining in under two minutes, and configuring Auto Scaling so capacity recovers without anyone stepping in.

#Confirm Spot interruptions are the cause

First, check whether instances were actually reclaimed. Scale-in, failed health checks and other events can look the same from the application side.

# Spot requests closed by an interruption in this Region
aws ec2 describe-spot-instance-requests \
  --filters Name=status-code,Values=instance-terminated-no-capacity,instance-terminated-capacity-oversubscribed,instance-terminated-by-price,instance-stopped-no-capacity \
  --query "SpotInstanceRequests[].{id:InstanceId,type:LaunchSpecification.InstanceType,az:LaunchedAvailabilityZone,code:Status.Code,time:Status.UpdateTime}" \
  --output table

CloudTrail records each interruption as a BidEvictedEvent:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=BidEvictedEvent \
  --start-time "$(date -u -d '-7 days' +%FT%TZ)" \
  --query "Events[].{time:EventTime,resources:Resources[].ResourceName}" --output json

For an Auto Scaling group, the activity history shows why each instance left:

aws autoscaling describe-scaling-activities --auto-scaling-group-name web-asg \
  --max-items 20 --query "Activities[].{time:StartTime,cause:Cause}" --output text

#The two signals

Use the rebalance recommendation to start slow work early, such as checkpointing or launching a replacement. Use the interruption notice as the hard deadline.

#Catch the notice on the instance

Both IMDS paths return 404 until a signal exists. Here is a small watcher that uses IMDSv2:

#!/usr/bin/env bash
# /usr/local/bin/spot-watch.sh - run as a systemd service
IMDS=http://169.254.169.254
while true; do
  TOKEN=$(curl -s -X PUT "$IMDS/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
  CODE=$(curl -s -o /tmp/spot-action -w '%{http_code}' \
    -H "X-aws-ec2-metadata-token: $TOKEN" "$IMDS/latest/meta-data/spot/instance-action")
  if [ "$CODE" = "200" ]; then
    logger -t spot-watch "Interruption notice: $(cat /tmp/spot-action)"
    /usr/local/bin/drain.sh
    exit 0
  fi
  sleep 5
done

The response body looks like {"action": "terminate", "time": "2026-10-31T14:02:00Z"}. The action can also be stop or hibernate, depending on the interruption behavior you chose.

#Drain in under two minutes

Keep drain.sh short and run its steps in order: stop taking new work, finish or hand off current work, then flush state.

#!/usr/bin/env bash
# /usr/local/bin/drain.sh
TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)
TG_ARN=arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/web/0123456789abcdef

aws elbv2 deregister-targets --target-group-arn "$TG_ARN" --targets Id="$ID"
systemctl stop my-worker        # stop pulling from the queue; unfinished SQS messages become visible again
sync                            # flush buffered writes

The instance role needs elasticloadbalancing:DeregisterTargets. Check the target group's deregistration delay. If it's longer than about 90 seconds, connections are still draining when the instance disappears:

aws elbv2 modify-target-group-attributes --target-group-arn "$TG_ARN" \
  --attributes Key=deregistration_delay.timeout_seconds,Value=60

Warning

Spot Instances can be terminated with only two minutes' notice, and instance store data is lost on termination. Never keep the only copy of anything on a Spot Instance. Write checkpoints and output to S3, EFS or a database while the job runs, not after the notice arrives.

For containers, let the platform do the drain. On ECS on EC2, set ECS_ENABLE_SPOT_INSTANCE_DRAINING=true in /etc/ecs/ecs.config so the agent sets the container instance to DRAINING. On EKS, Karpenter or the AWS Node Termination Handler cordons and drains the node.

#Catch the notice centrally with EventBridge

An EventBridge rule gives you a record of every interruption, and it fires even when the instance is too busy to react itself:

aws events put-rule --name spot-interruptions \
  --event-pattern '{"source":["aws.ec2"],"detail-type":["EC2 Spot Instance Interruption Warning","EC2 Instance Rebalance Recommendation"]}'

aws events put-targets --rule spot-interruptions \
  --targets Id=log,Arn=arn:aws:logs:eu-west-1:111122223333:log-group:/aws/events/spot

The event's detail contains instance-id and, for warnings, instance-action. Sending events to a log group lets you query interruption counts by instance type and AZ in CloudWatch Logs Insights. A CloudWatch Logs target also needs a resource policy that lets EventBridge write to the log group. The console adds it for you; with the CLI, use aws logs put-resource-policy.

#Configure Auto Scaling to recover on its own

A Spot-friendly Auto Scaling group has three things: many instance pools, the price-capacity-optimized allocation strategy and Capacity Rebalancing.

{
  "LaunchTemplate": {
    "LaunchTemplateSpecification": { "LaunchTemplateName": "web-lt", "Version": "$Latest" },
    "Overrides": [
      { "InstanceType": "m6i.large" }, { "InstanceType": "m5.large" },
      { "InstanceType": "m6a.large" }, { "InstanceType": "m7i.large" },
      { "InstanceType": "r6i.large" }
    ]
  },
  "InstancesDistribution": {
    "OnDemandBaseCapacity": 2,
    "OnDemandPercentageAboveBaseCapacity": 0,
    "SpotAllocationStrategy": "price-capacity-optimized"
  }
}

aws autoscaling update-auto-scaling-group --auto-scaling-group-name web-asg \
  --mixed-instances-policy file://mip.json \
  --capacity-rebalance \
  --vpc-zone-identifier "subnet-aaa,subnet-bbb,subnet-ccc"

• OnDemandBaseCapacity keeps a floor that Spot can't take away. Size it for the minimum load you must serve.
• Capacity Rebalancing starts a replacement when a rebalance recommendation arrives, then terminates the at-risk instance. Add a termination lifecycle hook if the drain needs more time than the instance gets by default.
• Instead of listing types, you can set InstanceRequirements (vCPU and memory ranges) in the overrides. This is attribute-based instance type selection, and it widens the pool list automatically.

Tip

Before you pick instance types, check how likely a request is to succeed. aws ec2 get-spot-placement-scores --instance-types m6i.large m5.large m6a.large --target-capacity 20 --region-names eu-west-1 --single-availability-zone scores each AZ from 1 to 10. A low score means you need more types or more AZs.

#Test it before EC2 does

AWS Fault Injection Service can send real interruption notices to your instances. This tests the watcher, the drain and the Auto Scaling replacement from start to finish:

{
  "description": "Interrupt 1 Spot instance in web-asg",
  "roleArn": "arn:aws:iam::111122223333:role/fis-spot-role",
  "stopConditions": [{ "source": "none" }],
  "targets": {
    "SpotInstances": {
      "resourceType": "aws:ec2:spot-instance",
      "resourceTags": { "aws:autoscaling:groupName": "web-asg" },
      "filters": [{ "path": "State.Name", "values": ["running"] }],
      "selectionMode": "COUNT(1)"
    }
  },
  "actions": {
    "interrupt": {
      "actionId": "aws:ec2:send-spot-instance-interruptions",
      "parameters": { "durationBeforeInterruption": "PT2M" },
      "targets": { "SpotInstances": "SpotInstances" }
    }
  }
}

TID=$(aws fis create-experiment-template --cli-input-json file://spot-test.json --query experimentTemplate.id --output text)
aws fis start-experiment --experiment-template-id "$TID"

While the experiment runs, watch three things: the instance logs spot-watch, the target leaves the target group, and the Auto Scaling group launches a replacement before or right after termination.

#Prevent it next time

• Use at least five or six instance types across three AZs for each Spot group, with price-capacity-optimized.
• Turn on Capacity Rebalancing and keep an On-Demand base for the minimum load you must serve.
• Run a watcher on every Spot instance, or use the platform's drain support (ECS agent draining, Karpenter or Node Termination Handler).
• Keep the deregistration delay and drain time under two minutes. Checkpoint long jobs as they run.
• Log every interruption with EventBridge and review counts by type and AZ each month.
• Run the FIS experiment after every change to the launch template or drain logic.