EBS volume attached but not visible in the OS: NVMe names, stuck attaching and fstab boot failures

#Symptom

You attached an EBS volume as /dev/sdf, the console shows it in-use, but one of these happens:

• There is no /dev/sdf or /dev/xvdf on the instance.
• The disk is there but mount fails, or the data looks empty.
• The volume sits in attaching and never finishes.
• After a reboot the instance drops into emergency mode or fails status checks because of /etc/fstab.
• On Windows the disk shows in Disk Management as Offline.

Work through the checks in order. The first two take a minute and rule out the most common causes.

#Quick map: symptom to cause

#Check 1: what AWS thinks is attached

VOL=vol-0abc123def4567890
INST=i-0123456789abcdef0

aws ec2 describe-volumes --volume-ids $VOL \
  --query "Volumes[].{State:State,AZ:AvailabilityZone,Type:VolumeType,Size:Size,Attach:Attachments[].{Inst:InstanceId,Dev:Device,State:State}}" \
  --output json

aws ec2 describe-instances --instance-ids $INST \
  --query "Reservations[].Instances[].{AZ:Placement.AvailabilityZone,Type:InstanceType,BDM:BlockDeviceMappings[].{Dev:DeviceName,Vol:Ebs.VolumeId,State:Ebs.Status}}" \
  --output json

Look for:

• The volume's AvailabilityZone must match the instance's. EBS volumes can't attach across AZs. Create a snapshot and restore it into the right AZ.
• The attachment State should be attached. If it's been attaching for more than a few minutes, go to Check 3.
• The device name you chose shouldn't already appear in the instance's block device mappings.
• A volume that's in-use on another instance can't be attached elsewhere unless it's io1/io2 with Multi-Attach enabled.

Every instance type also has a limit on how many volumes it can attach. If you've hit it, the attach call fails. Check the current limits for your instance type in the EC2 documentation.

#Check 2: find the disk by volume ID, not device name

On Nitro instances, EBS volumes appear as NVMe devices (/dev/nvme0n1, /dev/nvme1n1...). The kernel numbers them in the order they respond at boot, so the number can change between reboots and has no link to the /dev/sdf you asked for.

The volume ID is set as the NVMe serial number (without the hyphen), so use that:

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,SERIAL
# nvme1n1  100G disk              vol0abc123def4567890

On Amazon Linux, ebsnvme-id also shows the block device mapping name, and udev creates a /dev/sdf symlink:

sudo /sbin/ebsnvme-id /dev/nvme1n1
ls -l /dev/sdf

On other distributions, install nvme-cli:

sudo nvme id-ctrl -V /dev/nvme1n1 | grep -E '^sn|^mn'

On Windows, the ebsnvme-id.exe tool ships with recent AWS Windows AMIs and the AWS NVMe driver package:

& "C:\ProgramData\Amazon\Tools\ebsnvme-id.exe"
Get-Disk | Select-Object Number, FriendlyName, SerialNumber, OperationalStatus, Size

If the volume ID doesn't appear anywhere, the OS hasn't seen the device. Check dmesg | grep -i nvme on Linux. On Windows, check that the AWS NVMe driver is installed. Older custom AMIs migrated from Xen instance types sometimes lack the NVMe or ENA drivers.

#Check 3: volume stuck in attaching

The usual cause is that the device name you picked is already in use, either by the OS or by an earlier mapping. AWS's guidance is to detach and re-attach with a different device name.

aws ec2 detach-volume --volume-id $VOL --force
aws ec2 wait volume-available --volume-ids $VOL
aws ec2 attach-volume --volume-id $VOL --instance-id $INST --device /dev/sdg
aws ec2 wait volume-in-use --volume-ids $VOL

Caution

--force detaches without flushing the filesystem cache. Only use it when the volume is stuck in attaching or isn't mounted. On a volume that is mounted and taking writes, unmount it first or you can corrupt data.

If the volume is encrypted, look in CloudTrail for KMS AccessDenied errors around the attach time. If the key is disabled, or the instance's account can't use it, the attachment fails.

#Check 4: the disk is visible but won't mount

#Blank volume

A volume created empty has no filesystem:

sudo file -s /dev/nvme1n1        # "data" means no filesystem
sudo mkfs -t xfs /dev/nvme1n1
sudo mkdir -p /data && sudo mount /dev/nvme1n1 /data

Warning

Never run mkfs on a volume restored from a snapshot or moved from another instance. If file -s or lsblk -f shows a filesystem, the data is there; formatting erases it.

#Duplicate filesystem UUID

If you restore a snapshot of the root volume and attach it to the same instance (or a clone), both disks have the same filesystem UUID. XFS refuses to mount the second one, and dmesg shows Filesystem has duplicate UUID.

# XFS: mount once without the UUID check
sudo mount -o nouuid /dev/nvme1n1p1 /mnt/recovery

# XFS: or give it a new UUID permanently (filesystem must be unmounted and clean)
sudo xfs_admin -U generate /dev/nvme1n1p1

# ext4: new UUID
sudo e2fsck -f /dev/nvme1n1p1
sudo tune2fs -U random /dev/nvme1n1p1

Mount the partition (nvme1n1p1), not the whole disk, when the volume is partitioned. lsblk shows which.

#Duplicate LVM volume group

sudo pvs; sudo vgs                     # warnings about duplicate PVs or VG names
sudo vgimportclone --basevgname vg_restored /dev/nvme1n1p2
sudo vgchange -ay vg_restored

#Check 5: instance won't boot after adding the volume to fstab

An fstab entry that points at /dev/nvme1n1 breaks when the NVMe order changes. An entry for a volume that has been detached stops the boot unless it has nofail. Use the filesystem UUID and nofail for every non-root volume:

sudo blkid /dev/nvme1n1
# /etc/fstab
UUID=1b2c3d4e-....  /data  xfs  defaults,nofail  0  2
sudo mount -a && findmnt /data         # test before you reboot

If the instance is already stuck:

1. Use the EC2 serial console (it has to be enabled for the account, and the OS needs a user with a password) to get to the emergency shell, fix /etc/fstab, and reboot.
2. Or stop the instance, detach the root volume, attach it to a rescue instance in the same AZ, mount it (with nouuid if it's XFS from the same AMI), edit etc/fstab, then attach it back as the root device using the original root device name.

aws ec2 describe-instances --instance-ids $INST \
  --query "Reservations[].Instances[].RootDeviceName" --output text
aws ec2 get-console-output --instance-id $INST --latest --output text | tail -50

#Windows: disk is offline

Get-Disk | Where-Object OperationalStatus -eq 'Offline' |
  Select-Object Number, SerialNumber, Size, OfflineReason

Set-Disk -Number 1 -IsOffline $false
Set-Disk -Number 1 -IsReadOnly $false

# Only for a brand-new, empty volume:
Initialize-Disk -Number 1 -PartitionStyle GPT
New-Partition -DiskNumber 1 -UseMaximumSize -AssignDriveLetter |
  Format-Volume -FileSystem NTFS -NewFileSystemLabel Data

If OfflineReason is a signature collision (common when you attach a snapshot of another disk from the same server), bringing it online makes Windows write a new disk signature. That's fine for a data disk. Don't do it to a boot volume you plan to move back.

#Prevent it next time

• Refer to volumes by volume ID in scripts and by filesystem UUID or label in fstab. Never use /dev/nvmeXn1.
• Add nofail to every non-root fstab entry, and run mount -a before you reboot.
• Enable EC2 serial console access in your account ahead of time so it's ready when a server won't boot.
• When you restore a snapshot to inspect data, attach it to a separate recovery instance instead of the source server, so you avoid UUID and signature collisions.
• Keep NVMe and ENA drivers current in custom AMIs, especially Windows images built before Nitro.