Home / AWS

AWSTroubleshootingIntermediate

Sev1: EFS slow or mounts hanging — burst credits exhausted, I/O limit or network path

Fix applications that freeze on Amazon EFS. Check burst credit balance and PercentIOLimit, switch throughput mode, verify mount targets and security groups, and recover hung NFS mounts without rebooting everything.

Deepak Bhardwaj2 min read

Applies toAmazon EFS (General Purpose and Max I/O, Bursting, Provisioned and Elastic throughput), amazon-efs-utils, Linux NFS clients

Symptoms

  • Apps or containers hang on file operations; df hangs; processes stuck in D state.
  • Throughput collapses to a low baseline after a busy period.
  • New mounts time out on some instances but not others.

1. Throughput or network?

FS=fs-0123456789abcdef0
aws efs describe-file-systems --file-system-id $FS \
  --query "FileSystems[0].{mode:ThroughputMode, perf:PerformanceMode, sizeGiB:SizeInBytes.Value}"

for m in BurstCreditBalance PercentIOLimit MeteredIOBytes; do
  aws cloudwatch get-metric-statistics --namespace AWS/EFS --metric-name $m \
    --dimensions Name=FileSystemId,Value=$FS --start-time $(date -u -d '-6 hours' +%FT%TZ) \
    --end-time $(date -u +%FT%TZ) --period 300 --statistics Minimum Maximum --output table
done
Finding Cause
BurstCreditBalance near 0, Bursting mode Out of credits; throttled to baseline (scales with stored size)
PercentIOLimit near 100 (General Purpose) Hitting the I/O operations limit
Metrics fine, some clients can't mount Network: mount target, security group, DNS

2. Out of burst credits: switch to Elastic

aws efs update-file-system --file-system-id $FS --throughput-mode elastic

Elastic throughput scales with the workload and you pay for what you use. Provisioned throughput is the alternative for steady, predictable load. Check the current rules on how often you can change throughput mode.

Tip

Don't "fix" bursting by writing dummy data to grow the file system. Elastic mode is the supported answer.

3. Mount and network checks

aws efs describe-mount-targets --file-system-id $FS \
  --query "MountTargets[].{az:AvailabilityZoneName, ip:IpAddress, state:LifeCycleState, subnet:SubnetId}" --output table
aws efs describe-mount-target-security-groups --mount-target-id fsmt-0abc
  • A mount target in every AZ where clients run (cross-AZ mounts work but add cost and risk).
  • Mount target security group allows TCP 2049 from the client security group.
  • Clients resolve fs-xxxx.efs.<region>.amazonaws.com (VPC DNS resolution and hostnames enabled).

Recommended mount with efs-utils (TLS, resilient defaults):

sudo mount -t efs -o tls,_netdev fs-0123456789abcdef0:/ /mnt/efs

For plain NFS, use AWS-recommended options: nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport. noresvport helps reconnect after network events.

4. Recover a hung mount

sudo umount -f -l /mnt/efs     # lazy force unmount
sudo mount -a                  # remount from fstab

Processes stuck in D state may only clear after the mount recovers or the instance restarts. For EKS with the EFS CSI driver, restart the affected pods after the file system recovers.

Prevent it next time

  • Use Elastic throughput unless you have a clear reason not to.
  • Alarm on BurstCreditBalance (if bursting) and PercentIOLimit > 90%.
  • Keep small-file heavy workloads (package caches, build outputs) off EFS; use EBS or instance storage.

Related fixes