Home / AWS

AWSRunbookAdvanced

Sev1: EC2 root volume is 100% full. Services crash, SSH fails and Session Manager will not connect

A full root filesystem stops logging, databases and even SSH logins. Get in (or around the OS with a rescue instance), free space safely, grow the EBS volume online, and add alarms so it never fills silently again.

Deepak Bhardwaj4 min read

Applies toAmazon EC2 Linux (Amazon Linux 2023, Ubuntu, RHEL) with EBS root volumes; Windows notes included; AWS CLI v2, SSM

Symptoms

  • Application errors: No space left on device; databases stop accepting writes.
  • SSH connects then drops, or logins hang; Session Manager shows the agent as offline.
  • StatusCheckFailed_Instance may fire because the OS is unhealthy, while system checks pass.

Step 1: try to get in (5 minutes)

aws ssm start-session --target i-0abc1234def567890
aws ec2-instance-connect ssh --instance-id i-0abc1234def567890     # if EC2 Instance Connect is set up
aws ec2 get-console-output --instance-id i-0abc1234def567890 --latest --output text | tail -40

If any shell works, go to step 3. If not, step 2.

Step 2: grow the volume from outside (no shell needed)

EBS volumes can be enlarged while attached and in use. Extra space alone often lets the agent and sshd start working again.

VOL=$(aws ec2 describe-instances --instance-ids i-0abc1234def567890 \
  --query "Reservations[].Instances[].BlockDeviceMappings[?DeviceName=='/dev/xvda' || DeviceName=='/dev/sda1'].Ebs.VolumeId" --output text)
aws ec2 create-snapshot --volume-id $VOL --description "pre-resize sev1 $(date -u +%F)"
aws ec2 modify-volume --volume-id $VOL --size 60
aws ec2 describe-volumes-modifications --volume-ids $VOL --query "VolumesModifications[].[ModificationState,Progress]"

The OS does not see the new size until the partition and filesystem are grown (step 4), which needs a shell. If you still cannot get one, use a rescue instance:

  1. Stop the instance (snapshot taken already).
  2. Detach the root volume, attach it to a helper instance in the same AZ as /dev/sdf.
  3. On the helper: lsblk, mount the data partition (for XFS clones add -o nouuid), free space (step 3) and grow it (step 4).
  4. Detach, reattach to the original as its root device name, start.

Warning

Stopping an instance loses data on instance store volumes and changes a non-Elastic public IP. Check both before stopping.

Step 3: free space safely

Find what is big:

sudo df -h /
sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -15
sudo du -xh --max-depth=2 /var 2>/dev/null | sort -h | tail -15
sudo find / -xdev -type f -size +500M -printf '%s %p\n' 2>/dev/null | sort -n | tail -20

Deleted-but-open files still use space:

sudo lsof +L1 2>/dev/null | awk '{print $7, $9, $1, $2}' | sort -n | tail

Restart the process holding them (or truncate via /proc/<pid>/fd/<n>) to release the space.

Safe quick wins:

sudo journalctl --vacuum-size=200M
sudo dnf clean all || sudo apt-get clean
sudo find /var/log -name "*.gz" -mtime +7 -delete
sudo truncate -s 0 /var/log/app/huge.log      # truncate, do not rm, files a process has open
docker system prune -af 2>/dev/null           # only if Docker is used and images can be re-pulled

Caution

Never delete database files, /var/lib/rpm, or anything under /usr. If the big directory belongs to a database, move data or grow the disk instead.

Step 4: grow the partition and filesystem

lsblk
sudo growpart /dev/nvme0n1 1          # disk and partition number from lsblk
df -hT /                               # find the filesystem type
sudo xfs_growfs -d /                   # XFS (Amazon Linux, RHEL)
sudo resize2fs /dev/nvme0n1p1          # ext4 (Ubuntu)

Windows: after modify-volume, run in PowerShell:

$max = (Get-PartitionSupportedSize -DriveLetter C).SizeMax
Resize-Partition -DriveLetter C -Size $max

Step 5: confirm

  • df -h / below 80%.
  • Application, database and SSM agent healthy: aws ssm describe-instance-information --filters Key=InstanceIds,Values=i-0abc1234def567890.
  • Status checks green.

Prevent it next time

Install the CloudWatch agent with disk_used_percent and alarm on it:

aws cloudwatch put-metric-alarm --alarm-name web01-root-disk-85 \
  --namespace CWAgent --metric-name disk_used_percent \
  --dimensions Name=InstanceId,Value=i-0abc1234def567890 Name=path,Value=/ Name=device,Value=nvme0n1p1 Name=fstype,Value=xfs \
  --statistic Maximum --period 300 --evaluation-periods 2 --threshold 85 \
  --comparison-operator GreaterThanThreshold --alarm-actions $TOPIC

Dimensions must match exactly what the agent publishes; check with aws cloudwatch list-metrics --namespace CWAgent.

  • Logrotate for application logs, with size limits; SystemMaxUse= in journald.conf.
  • Put application data and logs on a separate EBS volume so the root disk cannot fill.
  • Data Lifecycle Manager snapshots, so a mistake during cleanup is recoverable.

Related fixes