Sev1: PostgreSQL data disk full — PANIC, crash and how to restart without losing data

#Symptoms

• Log lines like PANIC: could not write to file "pg_wal/xlogtemp.12345": No space left on device or ERROR: could not extend file ... No space left on device.
• PostgreSQL shut down and won't start, or stays up but every write fails.

#1. Do not delete files inside the data directory

Caution

Never delete files in pg_wal/, base/, pg_xact/ or global/ by hand, and never run pg_resetwal to "free space". Removing WAL that hasn't been checkpointed destroys the cluster's consistency. Free space outside those directories, or add space.

#2. See what is using space

df -h /var/lib/postgresql
sudo du -sh /var/lib/postgresql/16/main/* | sort -h
sudo du -sh /var/lib/postgresql/16/main/base/pgsql_tmp 2>/dev/null
sudo du -sh /var/log/postgresql

#3. Free space safely

Fastest options, in order:

1. Extend the volume (cloud disks can grow online) and the filesystem:

sudo growpart /dev/nvme1n1 1 && sudo resize2fs /dev/nvme1n1p1   # ext4
sudo xfs_growfs /var/lib/postgresql                              # xfs

2. Remove or move old server logs and unrelated files on the same volume (old dumps, core files).

3. If the server is stopped and base/pgsql_tmp is large, those temp files are safe to clear (they are removed at startup anyway).

#4. Start PostgreSQL and let it recover

sudo systemctl start postgresql@16-main
sudo tail -f /var/log/postgresql/postgresql-16-main.log

It replays WAL from the last checkpoint. Wait for database system is ready to accept connections.

#5. Fix what filled the disk

WAL piling up because of a slot or archiving:

SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots ORDER BY 3 DESC;

SELECT archived_count, failed_count, last_failed_wal, last_failed_time FROM pg_stat_archiver;

Drop a dead slot (its consumer will need re-syncing):

SELECT pg_drop_replication_slot('old_cdc_slot');

Fix archive_command failures; WAL is removed only after successful archiving.

Temp file explosion: find the query and cap temp usage:

ALTER SYSTEM SET temp_file_limit = '20GB';
SELECT pg_reload_conf();

Data growth: find the biggest tables:

SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC LIMIT 10;

#6. Guard rails

-- Stop one slot from filling the disk again (PostgreSQL 13+)
ALTER SYSTEM SET max_slot_wal_keep_size = '50GB';
SELECT pg_reload_conf();

A slot that exceeds this limit is invalidated instead of filling the disk.

#Prevent it next time

• Alerts at 70% and 85% disk use on data and WAL volumes.
• Put logs on a separate volume; consider WAL on its own volume.
• Set max_slot_wal_keep_size and alert on inactive slots.
• Keep a small "ballast" file on the volume that you can delete to buy minutes in an emergency.