Sev1: MySQL writes hang because the disk filled with binary logs

#Symptom

• Applications time out on writes; reads still work.
• SHOW PROCESSLIST shows many threads in states like waiting for handler commit or Waiting for disk space.
• Error log:

[ERROR] [MY-000035] [Server] Disk is full writing './binlog.004211' (OS errno 28 - No space left on device). Waiting for someone to free space...

MySQL waits rather than failing, so it resumes by itself once there's space.

#Step 1: confirm what fills the disk

df -h /var/lib/mysql
du -sh /var/lib/mysql/binlog.* | tail -5
ls /var/lib/mysql/binlog.* | wc -l

SHOW BINARY LOGS;
SELECT @@binlog_expire_logs_seconds, @@max_binlog_size;

If something else fills the disk (huge ibtmp1, a runaway general log, a backup written to the data volume), deal with that instead.

#Step 2: find the oldest binlog anything still needs

Replicas:

-- on each replica
SHOW REPLICA STATUS\G     -- Source_Log_File = the binlog it's currently reading

CDC tools (Debezium, DMS, Fivetran) also read binlogs. Check each connector's current position in its status or offsets.

The oldest file among all readers is your limit. Everything before it can go.

#Step 3: purge safely

PURGE BINARY LOGS TO 'binlog.004150';      -- deletes everything before this file
-- or by time
PURGE BINARY LOGS BEFORE NOW() - INTERVAL 2 DAY;

PURGE BINARY LOGS refuses to delete a log currently being read by a connected replica, but it doesn't know about disconnected replicas or CDC tools. That's why you check Step 2 first.

Caution

Never rm binlog files at the OS level. The binlog index (binlog.index) then points to missing files, and replication and PURGE can fail. If you're truly at zero bytes and MySQL can't run PURGE, free space elsewhere first (move an old backup, extend the volume).

If you can't purge enough (a replica is days behind), the options are:

1. Extend the volume (fastest on cloud disks; most filesystems grow online).
2. Accept rebuilding the lagging replica / re-snapshotting CDC, and purge past it.

#Step 4: confirm recovery

SHOW PROCESSLIST;     -- waiting threads clear
SHOW BINARY LOGS;

Check applications recovered; restart pools if connections timed out.

#Step 5: fix retention

SET PERSIST binlog_expire_logs_seconds = 259200;     -- 3 days; size for your PITR and replica needs

Size the binlog volume as: daily binlog volume × retention days × 1.5, plus headroom.

Separate binlogs onto their own volume (log_bin = /mysql-binlog/binlog) so a binlog surge doesn't stop the data files from growing, and vice versa.

#Prevent it next time

• Disk alerts at 75% and 85%, plus a growth-rate alert.
• Alert on replica lag and stopped CDC connectors. They're the usual reason binlogs can't be purged.
• Storage autogrow where the platform supports it.
• Large batch jobs (bulk updates) chunked, so they don't generate huge binlog bursts.