PostgreSQL pg_wal filling the disk: inactive replication slots and failed archiving

#The symptom

Disk usage on the database volume climbs steadily, and almost all of it is in pg_wal. Checkpoints run, but the WAL never shrinks. If nobody acts, the volume fills and PostgreSQL stops with something like:

PANIC:  could not write to file "pg_wal/xlogtemp.12345": No space left on device

PostgreSQL removes or recycles a WAL segment at checkpoint only when nothing still needs it. Something is holding WAL back. In practice it is one of these:

Caution

Never delete files from pg_wal by hand, and never run pg_resetwal to "clean up". Removing a segment that is still needed corrupts the cluster or breaks crash recovery. Release whatever is holding WAL and let a checkpoint remove the segments.

#1. Confirm WAL is what is growing

On the server:

df -h /var/lib/postgresql
du -sh "$PGDATA"/pg_wal
ls "$PGDATA"/pg_wal | wc -l

From SQL (needs superuser or the pg_monitor role):

SELECT count(*) AS segments,
       pg_size_pretty(sum(size)) AS wal_size
FROM pg_ls_waldir();

SHOW max_wal_size;
SHOW wal_keep_size;
SHOW max_slot_wal_keep_size;

If wal_size is far above max_wal_size plus wal_keep_size, WAL is being retained, not just generated.

On Amazon RDS, watch the CloudWatch metrics TransactionLogsDiskUsage, OldestReplicationSlotLag and FreeStorageSpace:

aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS --metric-name OldestReplicationSlotLag \
  --dimensions Name=DBInstanceIdentifier,Value=prod-pg \
  --start-time "$(date -u -d '-6 hours' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 300 --statistics Maximum

#2. Check replication slots

SELECT slot_name, slot_type, database, active, active_pid,
       wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal,
       pg_size_pretty(safe_wal_size) AS safe_wal_size
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC NULLS LAST;

How to read it:

• retained_wal is how much WAL this slot alone forces the server to keep. The largest one is your culprit.
• active = f means nothing is connected to the slot. It still holds WAL indefinitely.
• wal_status is reserved or extended when healthy, unreserved when the slot is about to lose WAL, and lost when the WAL it needs is already gone and the slot is useless.
• safe_wal_size is only set when max_slot_wal_keep_size is set; it shows how much more WAL can be written before the slot is invalidated.

On PostgreSQL 17 and later, inactive_since shows when the slot last disconnected, and invalidation_reason shows why a slot was invalidated.

Find out who owns the slot before you touch it. Slot names usually give it away (debezium, dms_..., pg_standby_1, fivetran_...). For physical slots, match against the standbys you expect:

SELECT pid, application_name, client_addr, state,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn)) AS replay_lag
FROM pg_stat_replication;

#3. Check the archiver

If archive_mode = on, a segment cannot be recycled until archive_command (or archive_library) reports success.

SELECT archived_count, last_archived_wal, last_archived_time,
       failed_count, last_failed_wal, last_failed_time
FROM pg_stat_archiver;

last_failed_time newer than last_archived_time means archiving is broken right now. Count the backlog:

ls "$PGDATA"/pg_wal/archive_status/*.ready 2>/dev/null | wc -l

The server log has the real error:

grep -E "archive command failed|archiver" /var/log/postgresql/postgresql-*.log | tail -20

Typical causes are expired credentials or a changed bucket policy for the archive target, a full NFS mount, or a backup tool upgrade that changed the command path. Run the command by hand as the postgres user with a real segment name to see the error. For pgBackRest:

sudo -u postgres pgbackrest --stanza=main check

Once the command succeeds again, the archiver works through the .ready files and the next checkpoints shrink pg_wal. Do not set archive_command to /bin/true to clear the backlog unless you accept a gap in your point-in-time recovery chain and take a new full backup straight after.

#4. Release the slot

If the consumer should still exist, fix it first: restart the Debezium connector, resume the DMS task, or bring the standby back. A connected consumer advances the slot and WAL drains without data loss for that consumer.

If the consumer is gone or cannot catch up in time, drop the slot:

-- If the slot is active, stop the connection that holds it first
SELECT pg_terminate_backend(active_pid)
FROM pg_replication_slots
WHERE slot_name = 'debezium_orders' AND active;

SELECT pg_drop_replication_slot('debezium_orders');

Warning

Dropping a slot is not reversible. A logical consumer (CDC, logical replica) loses its position and needs a fresh snapshot or full resync. A physical standby that used the slot will fail with requested WAL segment ... has already been removed and must be rebuilt with pg_basebackup, unless it can fetch WAL from the archive. Agree this with the owner of the consumer.

Then force a checkpoint and confirm space comes back:

CHECKPOINT;
SELECT pg_size_pretty(sum(size)) FROM pg_ls_waldir();

Segments are removed or recycled at the checkpoint. It can take one or two checkpoints for the size to settle.

On RDS and Azure Flexible Server you cannot see the file system, but pg_drop_replication_slot and CHECKPOINT work for the admin role, and the storage metric drops after a few minutes.

#If the disk is already full and PostgreSQL is down

1. Do not delete anything in pg_wal.
2. Add space: grow the cloud disk and file system (growpart and resize2fs or xfs_growfs), or extend the LVM volume. On RDS, increase allocated storage or rely on storage autoscaling; on Azure Flexible Server, increase storage size.
3. If you cannot grow the volume quickly, free space elsewhere on the same file system: old server logs in log/, core dumps, stray backup files. Stop there.
4. Start PostgreSQL, then do steps 2 to 4 above before the space runs out again.

#Prevent it next time

Cap how much WAL a slot may hold. A slot that exceeds the cap is invalidated instead of filling the disk:

ALTER SYSTEM SET max_slot_wal_keep_size = '50GB';
SELECT pg_reload_conf();

Pick a value that fits comfortably inside your free space. The trade-off: a consumer that falls behind by more than the cap loses its slot and needs a resync. That is better than a primary outage.

On PostgreSQL 18, you can also invalidate slots that sit unused for too long. It is off by default (0) and is checked at checkpoint time:

ALTER SYSTEM SET idle_replication_slot_timeout = '1d';
SELECT pg_reload_conf();

Set it longer than any planned pause of your CDC pipelines or standby maintenance.

On managed services, set both parameters in the DB parameter group (RDS) or server parameters (Azure Flexible Server). Check that your engine version exposes them.

Alerting checklist:

• Alert on pg_wal size, or TransactionLogsDiskUsage on RDS, above a fixed threshold, not only on overall disk usage.
• Alert when any slot's retained WAL exceeds a threshold, or when a slot stays active = f for more than an hour.
• Alert when pg_stat_archiver.last_failed_time is newer than last_archived_time.
• Drop slots as part of decommissioning a standby or CDC pipeline. Abandoned slots from finished migrations are the most common cause of this incident.
• Keep at least the cap plus max_wal_size free on the WAL volume.

Related: streaming replication lag and pgBackRest point-in-time restore.