Sev1: EBS volume impaired, stuck attaching or I/O hung — recover the data volume fast

#Symptoms

• Database or app processes stuck in D state on Linux; dmesg shows I/O timeouts on nvme devices.
• Volume status check impaired or io-enabled: false.
• Volume stuck in attaching or detaching after an instance replacement.

#1. Check volume status

aws ec2 describe-volume-status --volume-ids vol-0abc \
  --query "VolumeStatuses[].{status:VolumeStatus.Status, details:VolumeStatus.Details, events:Events, actions:Actions}"
aws ec2 describe-volumes --volume-ids vol-0abc \
  --query "Volumes[].{state:State, attach:Attachments, type:VolumeType, iops:Iops, tput:Throughput}"

#2. Volume I/O disabled

When EBS detects a potential inconsistency it disables I/O. You choose when to re-enable, after you are ready to run a consistency check.

aws ec2 enable-volume-io --volume-id vol-0abc

Then check the filesystem (unmounted) and the application's own consistency checks:

sudo umount /data
sudo xfs_repair -n /dev/nvme1n1     # dry run first for XFS
sudo fsck -n /dev/nvme1n1           # ext4 dry run

Caution

Take a snapshot before any repair that writes to the volume. aws ec2 create-snapshot --volume-id vol-0abc --description "pre-repair INC12345".

To auto-enable I/O on future events (useful only for workloads that tolerate it):

aws ec2 modify-volume-attribute --volume-id vol-0abc --auto-enable-io

#3. Stuck attaching or detaching

Common causes: the device name is already in use on the instance, or the instance OS still holds the volume.

# Try a normal detach first
aws ec2 detach-volume --volume-id vol-0abc
# Then force only if still stuck
aws ec2 detach-volume --volume-id vol-0abc --force

Warning

--force detaches without letting the OS flush caches, like pulling a disk. Use it only when the instance is unrecoverable or stopped, and run filesystem and database checks afterwards.

Reattach with a free device name:

aws ec2 attach-volume --volume-id vol-0abc --instance-id i-0new --device /dev/sdf

If it still won't attach, stop and start the target instance (not reboot), which moves it to new hardware, then attach again.

#4. Performance saturation that looks like a hang

aws cloudwatch get-metric-statistics --namespace AWS/EBS --metric-name VolumeQueueLength \
  --dimensions Name=VolumeId,Value=vol-0abc --statistics Average --period 60 \
  --start-time "$(date -u -d '-1 hour' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)"

Also check VolumeThroughputPercentage, VolumeIOPSExceededCheck and VolumeThroughputExceededCheck (newer metrics, check availability for your volume type), plus the instance's EBS bandwidth limit.

Raise performance online:

aws ec2 modify-volume --volume-id vol-0abc --volume-type gp3 --iops 12000 --throughput 500

#5. Restore from snapshot

If the volume is unrecoverable:

aws ec2 describe-snapshots --filters Name=volume-id,Values=vol-0abc \
  --query "reverse(sort_by(Snapshots,&StartTime))[:3].{id:SnapshotId, t:StartTime}" --output table
aws ec2 create-volume --snapshot-id snap-0latest --availability-zone eu-west-1a --volume-type gp3

Volumes created from snapshots load blocks lazily; first reads are slow. Use Fast Snapshot Restore for critical volumes or pre-warm with fio/dd reads.

#Prevent it next time

• Snapshot schedules with AWS Backup or Data Lifecycle Manager for every production data volume.
• Alarms on volume status checks and queue length.
• Mount with nofail in /etc/fstab so a missing data volume does not stop the boot.
• Size gp3 IOPS and throughput for peak, not average.