Redis MISCONF: writes refused after a failed RDB snapshot, restore writes safely

#Symptom

Reads work, but writes fail with an error like this. The wording varies by version:

MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist to disk.
Commands that may modify the data set are disabled...

PING also returns the error on the affected server, so health checks can report Redis as down even though it is still serving reads.

Redis does this on purpose. It refuses writes when all three of these are true:

• stop-writes-on-bgsave-error is yes (the default).
• At least one save point is configured.
• The last background save failed.

Writes resume automatically once a background save succeeds. You usually don't need to restart anything.

Caution

Don't restart Redis to clear this error. With RDB-only persistence, a restart reloads the last good snapshot, and everything written since then is gone. If the data matters, find the cause and get a good snapshot first. Avoid SHUTDOWN NOSAVE for the same reason.

#Step 1: Confirm which failure you have

Add -h, -p and credentials as needed. redis-cli reads a password from the REDISCLI_AUTH environment variable.

redis-cli INFO persistence | grep -E 'rdb_last_bgsave_status|rdb_bgsave_in_progress|rdb_last_save_time|rdb_changes_since_last_save|aof_enabled|aof_last_write_status|aof_last_bgrewrite_status'
redis-cli CONFIG GET stop-writes-on-bgsave-error
redis-cli CONFIG GET save
redis-cli CONFIG GET dir
redis-cli CONFIG GET dbfilename
redis-cli CONFIG GET logfile

• rdb_last_bgsave_status:err means this runbook applies.
• rdb_last_save_time is the Unix time of the last good snapshot. Convert it with date -d @<value>. It shows how much data is at risk.
• rdb_changes_since_last_save shows how many changes that snapshot doesn't include.
• aof_last_write_status:err, with the message MISCONF Errors writing to the AOF file, means the failure is in the AOF path. See Step 4.

The status stays err until a save succeeds, so the original fault may be hours old and may already have gone away.

#Step 2: Find the cause in the log

Use the logfile path from Step 1. If it is empty, Redis logs to standard output, which usually means journald or the container log. Service names vary by distribution.

sudo grep -iE "background sav|can't save|error.*(rdb|db on disk)|failed opening" /var/log/redis/redis-server.log | tail -20
sudo journalctl -u redis-server --since "2 hours ago" | grep -iE "background sav|can't save|rdb"

Then check the host:

DIR=$(redis-cli --raw CONFIG GET dir | tail -1)
df -h "$DIR"; df -i "$DIR"
ls -ld "$DIR"; ls -lh "$DIR"
findmnt -no OPTIONS -T "$DIR"
free -m
sysctl vm.overcommit_memory
sudo dmesg -T | grep -iE 'out of memory|killed process' | tail
redis-cli INFO memory | grep -E 'used_memory_rss_human|maxmemory_human'

#Step 3: Decide whether to restore writes before the fix

If the cause will take only minutes to fix, go straight to Step 4.

If the application is down and the fix will take longer, you can let writes through while you work:

redis-cli CONFIG SET stop-writes-on-bgsave-error no

Warning

This only hides the failure. Snapshots are still failing, so anything written from now on is lost if Redis stops before a save succeeds. Record the change in the incident notes, and set it back in Step 5.

#Step 4: Fix the cause

#Fix A: disk or inodes full

The child writes a temporary file and renames it over dump.rdb when it finishes. For that reason, you need room for the new snapshot while the old one still exists.

Grow the volume or remove unrelated files. Look for leftover temporary files only when no save is running:

redis-cli INFO persistence | grep rdb_bgsave_in_progress   # must be 0
ls -lh "$DIR"/temp-*.rdb

Caution

Don't delete dump.rdb. Until a new save succeeds, it may be your only copy of the data on disk. Delete a temp-*.rdb file only when rdb_bgsave_in_progress is 0.

From Redis 7.0, CONFIG SET dir is refused unless enable-protected-configs was set at startup, so don't plan on moving saves to another volume at runtime.

#Fix B: permissions or read-only file system

Test as the account Redis runs under:

ps -o user= -C redis-server
sudo -u redis touch "$DIR/.write-test" && sudo -u redis rm "$DIR/.write-test"
systemctl cat redis-server | grep -E 'ReadWritePaths|ProtectSystem'

If the owner is wrong, fix the owner of the directory: sudo chown redis:redis "$DIR". Files inside it may need the same fix.

If findmnt shows ro, the kernel may have remounted the volume after an I/O error. Check dmesg and treat that as a storage incident. Don't just remount it.

If the service unit has ProtectSystem and dir isn't in ReadWritePaths, add it with a drop-in. That change needs a service restart, so plan it after you have a good snapshot.

#Fix C: fork refused

The Redis admin guide recommends vm.overcommit_memory = 1. Set it now, and persist it:

sudo sysctl vm.overcommit_memory=1
echo 'vm.overcommit_memory = 1' | sudo tee /etc/sysctl.d/99-redis.conf

#Fix D: child killed by the OOM killer

During a save, every page the parent changes gets copied, so a write-heavy instance needs memory well beyond its dataset size. rdb_last_cow_size shows how much the last successful save used.

• Run the save during a quieter period.
• Add host memory, or raise the container's memory limit.
• Disable Transparent Huge Pages, which the admin guide also recommends: echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled.

#AOF write errors

If the error names the AOF file, the cause is usually the same disk, permission or read-only problem as above. A manual BGSAVE won't clear it. Fix the disk, then confirm that aof_last_write_status returns to ok. If aof_last_bgrewrite_status:err is also set, run redis-cli BGREWRITEAOF once there is room.

#Step 5: Take a snapshot and verify

redis-cli BGSAVE SCHEDULE
redis-cli INFO persistence | grep -E 'rdb_bgsave_in_progress|rdb_last_bgsave_status|rdb_last_bgsave_time_sec'
redis-cli LASTSAVE

SCHEDULE waits for a save already in progress instead of returning an error. Repeat the INFO command until rdb_bgsave_in_progress is 0 and rdb_last_bgsave_status is ok, then test a write:

redis-cli SET misconf:test ok EX 60
redis-cli GET misconf:test

If you changed the setting in Step 3, set it back:

redis-cli CONFIG SET stop-writes-on-bgsave-error yes

#Prevent it next time

Alert before clients get errors. This check can run from cron or your monitoring agent, using a threshold based on your save settings:

#!/usr/bin/env bash
MAX_AGE=7200   # seconds; set from your save points
INFO=$(redis-cli INFO persistence | tr -d '\r')
STATUS=$(echo "$INFO" | awk -F: '/^rdb_last_bgsave_status/{print $2}')
LAST=$(echo "$INFO" | awk -F: '/^rdb_last_save_time/{print $2}')
AGE=$(( $(date +%s) - LAST ))
if [ "$STATUS" != "ok" ] || [ "$AGE" -gt "$MAX_AGE" ]; then
  echo "CRITICAL: bgsave status=$STATUS, last good save ${AGE}s ago"; exit 2
fi
echo "OK: last good save ${AGE}s ago"

• Alert on rdb_last_bgsave_status, aof_last_write_status and snapshot age. Don't rely on PING for this.
• Alert on disk space and inodes for the dir volume, and size it to hold two snapshots.
• Set vm.overcommit_memory = 1 and disable Transparent Huge Pages in your host build.
• Leave at least rdb_last_cow_size of memory free on the host.
• Copy snapshots off the host, and test a restore.