Home / AWS

AWSTroubleshootingIntermediate

Slow disk on EC2: troubleshooting EBS volume performance

Find out whether you're hitting the volume's limits, the instance's EBS bandwidth or burst credits, then fix it with an online gp3 change.

Deepak Bhardwaj3 min read

Applies toEBS gp2/gp3/io2 volumes, Nitro instances, AWS CLI v2

A slow disk on EC2 has one of three causes: the volume is at its IOPS or throughput limit, the instance is at its EBS bandwidth limit, or the workload is bursting and has run out of credits. Each one shows up in a different metric.

1. Look from inside the guest

iostat -xz 5          # Linux: watch r/s, w/s, rMB/s, wMB/s, r_await, w_await, aqu-sz

On Windows, use Performance Monitor: PhysicalDisk\Avg. Disk sec/Read, Disk Transfers/sec and Current Disk Queue Length.

High latency (await above about 10 ms on SSD volumes) together with IOPS sitting flat at a round number usually means you've hit a cap.

2. Volume-level metrics

VOL=vol-0abc123
aws cloudwatch get-metric-statistics --namespace AWS/EBS --metric-name VolumeQueueLength \
  --dimensions Name=VolumeId,Value=$VOL --statistics Average \
  --start-time "$(date -u -d '-3 hours' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" --period 300
Metric What to look for
VolumeReadOps + VolumeWriteOps (÷ period) Close to the provisioned IOPS → volume IOPS cap
VolumeReadBytes + VolumeWriteBytes Close to the provisioned throughput → throughput cap
VolumeIOPSExceededCheck / VolumeThroughputExceededCheck 1 means the workload asked for more than the volume provides
BurstBalance (gp2, st1, sc1) Falling to 0 → credits exhausted, back to baseline
VolumeQueueLength Constantly high → requests are queuing

3. Instance-level limits

Every instance type has a maximum EBS bandwidth and IOPS, shared by all attached volumes. Smaller sizes can only burst to that maximum for a limited time.

aws ec2 describe-instance-types --instance-types m6i.large \
  --query "InstanceTypes[].EbsInfo.EbsOptimizedInfo" --output table

Check EBSIOBalance% and EBSByteBalance% in the AWS/EC2 namespace. If they drop toward 0, the instance is the bottleneck, and adding more volumes won't help. Use a larger instance size.

4. Fix: modify the volume online

gp3 lets you set IOPS and throughput independently of size, and the change is applied without downtime:

aws ec2 modify-volume --volume-id $VOL --volume-type gp3 --iops 6000 --throughput 500
aws ec2 describe-volumes-modifications --volume-ids $VOL \
  --query "VolumesModifications[].{state:ModificationState,progress:Progress}" --output table

Important

After a modification you must wait 6 hours before you can modify the same volume again. Pick numbers with some headroom, not just enough for today.

Moving from gp2 to gp3 is usually a win. gp3 has a 3,000 IOPS / 125 MiB/s baseline regardless of size and no burst credits to run out of.

5. Other common causes

  • Volumes restored from snapshots are slow on first read, because blocks load lazily from S3. Pre-warm them with fio or dd across the whole device, or enable Fast Snapshot Restore on critical snapshots.
  • A single-threaded workload with queue depth 1 can't use high IOPS. Latency is your limit, not provisioning.
  • RAID 0 across several gp3 volumes raises the volume ceiling, but not the instance limit.

Alarms worth having

  • BurstBalance < 20 on any remaining gp2 volumes.
  • VolumeIOPSExceededCheck or VolumeThroughputExceededCheck at 1 for 15 minutes.
  • EBSIOBalance% < 20 on burstable-EBS instance sizes.

Related fixes