EBS volume restored from a snapshot is slow: lazy loading, Fast Snapshot Restore and pre-warming

#Symptom

After restoring a database server or rebuilding from a snapshot in a DR test:

• The volume is attached and the instance boots, but the application is very slow for hours.
• iostat shows high await on the restored volume even though IOPS are far below the provisioned gp3/io2 limits.
• CloudWatch VolumeTotalReadTime / VolumeReadOps show long average read latency.
• A full DR test misses its RTO even though "the restore took 2 minutes".

#Why it happens

A new volume from a snapshot is available immediately, but its blocks are still in Amazon S3. Each block is fetched the first time it is read. That first read is much slower than a normal EBS read. After a block has been read once, it performs normally.

So the restore is fast, but the volume is not "warm". Databases that touch most of their data files (crash recovery, index scans, backups) pay the penalty on every block.

#Step 1: confirm it's initialization, not something else

# Per-volume latency (ms) over the last hour
VOL=vol-0abc
aws cloudwatch get-metric-statistics --namespace AWS/EBS --metric-name VolumeTotalReadTime \
  --dimensions Name=VolumeId,Value=$VOL --start-time $(date -u -d '-1 hour' +%FT%TZ) \
  --end-time $(date -u +%FT%TZ) --period 300 --statistics Sum

Divide VolumeTotalReadTime by VolumeReadOps for the same period to get average seconds per read. Then check:

• Is the volume new (created from a snapshot in the last few hours)? aws ec2 describe-volumes --volume-ids $VOL --query "Volumes[].{created:CreateTime,snap:SnapshotId}"
• Are you hitting the provisioned IOPS/throughput or the instance's EBS bandwidth? If yes, it's sizing, not initialization: raise the volume's IOPS/throughput or move to an instance type with more EBS bandwidth.

#Fix A: pre-warm (initialize) the volume by reading every block

Read the whole device once before handing it to the application.

Linux:

lsblk                       # find the device, e.g. /dev/nvme1n1
sudo fio --filename=/dev/nvme1n1 --rw=read --bs=1M --iodepth=32 \
  --ioengine=libaio --direct=1 --name=volume-initialize

Windows (from the fio for Windows build, or with dd for Windows): read the raw physical drive the same way, for example --filename=\\.\PhysicalDrive1.

Warning

Only ever read (--rw=read). A write test against the raw device destroys the file system.

Time needed depends on volume size and the initialization rate. Pre-warming a multi-terabyte volume can take hours, so it fits planned restores better than emergencies.

#Fix B: Fast Snapshot Restore (FSR)

FSR keeps a snapshot pre-initialized in chosen Availability Zones. Volumes created from it deliver full performance immediately.

aws ec2 enable-fast-snapshot-restores --availability-zones eu-west-1a eu-west-1b \
  --source-snapshot-ids snap-0123

aws ec2 describe-fast-snapshot-restores \
  --filters Name=snapshot-id,Values=snap-0123 \
  --query "FastSnapshotRestores[].{az:AvailabilityZone,state:State}"

Wait until the state is enabled. Things to know:

• FSR is billed per snapshot, per AZ, per hour while enabled. Turn it on for DR-critical snapshots, not for every daily backup.
• Volume creation from an FSR snapshot uses credits. Creating many volumes at once can fall back to normal (lazy) behaviour; check describe-fast-snapshot-restores and the current FSR credit rules.
• It must be enabled in the AZ where you create the volume.

#Fix C: provisioned volume initialization rate

AWS also lets you specify a volume initialization rate when you create the volume from a snapshot, so the whole volume is initialized in a predictable time without paying for FSR around the clock. Check availability and the supported range in your region.

aws ec2 create-volume --availability-zone eu-west-1a --snapshot-id snap-0123 \
  --volume-type gp3 --volume-initialization-rate 300

It is charged per volume creation based on size. Use it when you need a predictable "fully warm by X" time.

#Choosing

#RDS and other services

RDS, Aurora snapshots restored as new instances and EBS-backed AMIs behave the same way: data loads lazily. For RDS, run representative reads (or a full table scan of hot tables) after restore and before switching traffic.

#Prevent it next time

• Include "volume warm" as a step in your DR runbook, with a measured duration.
• Automate FSR enable/disable on the latest snapshot of critical volumes with Data Lifecycle Manager (it supports FSR in snapshot policies).
• Measure the RTO in DR tests until the app is fast, not just until it is up.