Home / AWS

AWSTroubleshootingIntermediate

EC2 status check failures: system vs instance, and what to do about each

A failed system check is AWS's problem and an instance check is yours. Here's how to tell them apart and recover each one.

Deepak Bhardwaj3 min read

Applies toAWS CLI v2, Nitro-based instances

EC2 runs two checks every minute, and each failure has a different owner:

Check What it tests Who fixes it
System status The physical host, power, network reachability AWS. You can move the instance to another host.
Instance status Your OS: kernel, network config, full disk, boot You
Attached EBS status Whether the attached EBS volumes are reachable and can do I/O Usually AWS. Check the volume status.

Confirm which check is failing

ID=i-0abc123def4567890
aws ec2 describe-instance-status --instance-ids $ID --include-all-instances \
  --query "InstanceStatuses[].{state:InstanceState.Name,system:SystemStatus.Status,instance:InstanceStatus.Status,events:Events}" \
  --output table

Scheduled events in Events, such as system-reboot or instance-retirement, mean AWS has already told you about a host problem.

System check failed

  1. Stop and start the instance (a reboot isn't enough). On EBS-backed instances this moves the instance to a new host.
    aws ec2 stop-instances --instance-ids $ID && aws ec2 wait instance-stopped --instance-ids $ID
aws ec2 start-instances --instance-ids $ID
  2. For the future, set up automatic recovery with a CloudWatch alarm on StatusCheckFailed_System that runs the recover action. Many current instance types also have simplified automatic recovery turned on by default.

Warning

A stop/start erases instance store volumes and changes the public IPv4 address unless you use an Elastic IP.

Instance check failed

Read the console before you reboot:

aws ec2 get-console-output --instance-id $ID --latest --output text | tail -60
aws ec2 get-console-screenshot --instance-id $ID --query ImageData --output text | base64 -d > screen.jpg

Common causes:

  • Kernel panic or a bad fstab entry: shown in the console output. Use the EC2 Serial Console, or the rescue-instance method below.
  • Full root volume: services fail and SSH disconnects right after login. Grow the volume (aws ec2 modify-volume), then grow the partition and filesystem.
  • Network misconfiguration: someone edited netplan/ifcfg. Use Session Manager if the SSM agent still reaches its endpoints.
aws ssm start-session --target $ID
# Or let AWS automation diagnose and fix common OS problems
aws ssm start-automation-execution --document-name "AWSSupport-ExecuteEC2Rescue" \
  --parameters "UnreachableInstanceId=$ID"

The rescue-instance method

If nothing else works, fix the root volume from another instance:

  1. Stop the broken instance and detach its root volume.
  2. Attach the volume to a healthy instance in the same AZ as a secondary device.
  3. Mount it, fix the problem (/etc/fstab, sshd_config, cloud-init, disk space), then unmount.
  4. Reattach it as the root device (/dev/xvda or /dev/sda1, matching the original) and start the instance.

Tip

Take a snapshot of the volume before you edit anything on it. It's quick and gives you a way back.

Alarms worth having

aws cloudwatch put-metric-alarm --alarm-name "$ID-system-check" \
  --namespace AWS/EC2 --metric-name StatusCheckFailed_System --dimensions Name=InstanceId,Value=$ID \
  --statistic Maximum --period 60 --evaluation-periods 2 --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold \
  --alarm-actions arn:aws:automate:us-east-1:ec2:recover

Related fixes