Redis replica stuck in a full resync loop: output buffer limits, backlog size and repl-timeout

#The symptom

A replica never settles. master_link_status flips between up and down, the primary forks again and again, and sync_full keeps climbing. While this goes on you may also see:

• Reads on the replica failing with LOADING Redis is loading the dataset in memory
• Writes on the primary failing with NOREPLICAS Not enough good replicas to write, if min-replicas-to-write is set

One full sync after a new replica joins or a replica crashes is normal. Repeated full syncs for the same replica are the problem. Each cycle costs a fork, a full transfer and a full load, and you have no working replica while it runs.

#Confirm the loop

Run these on the primary. Set REDISCLI_AUTH first if the server needs a password.

# Counters since the last restart: sample twice, a few minutes apart
redis-cli INFO stats | grep -E '^sync_(full|partial_ok|partial_err)'

# Role, replica state and backlog
redis-cli INFO replication

# Output buffer use per replica: watch omem and tot-mem grow
redis-cli CLIENT LIST TYPE replica

# Fork health
redis-cli INFO persistence | grep -E 'rdb_bgsave_in_progress|rdb_last_bgsave_status|rdb_last_cow_size'

On the replica:

redis-cli INFO replication | grep -E 'master_link_status|master_sync_in_progress|master_link_down_since_seconds|master_sync_perc'
redis-cli INFO persistence | grep -E '^loading'

Counter names can differ slightly by version, so check your build's output. master_sync_perc, for example, is missing on older builds.

#Find the cause in the logs

The logs on both nodes usually name the cause. Exact wording varies by version, so search for the key phrase.

# Default paths vary by package; check the logfile setting if nothing matches
grep -E 'output buffer limits|partial resync|Replication ID mismatch|timedout replica|Cannot allocate' /var/log/redis/redis-server.log | tail -50

# On the replica
grep 'Timeout receiving' /var/log/redis/redis-server.log | tail
dmesg -T | grep -iE 'out of memory|oom-kill' | tail

#Why the buffer limit causes a loop

During a full sync the primary keeps every new write for that replica in its output buffer. The buffer grows from the start of the RDB snapshot until the replica has finished loading it.

The default limit is replica 256mb 64mb 60. The connection is closed if the buffer passes 256 MB, or stays above 64 MB for 60 seconds. A large dataset takes minutes to transfer and load, so a moderate write rate can trip the limit every time. The replica reconnects, the backlog can't cover the gap, and the sync starts over.

#Size the limits from your numbers

Measure the replication write rate on the primary:

a=$(redis-cli INFO replication | tr -d '\r' | awk -F: '/^master_repl_offset/{print $2}')
sleep 60
b=$(redis-cli INFO replication | tr -d '\r' | awk -F: '/^master_repl_offset/{print $2}')
echo "$(( (b - a) / 60 )) bytes/sec"

Measure it at peak, and also while the batch jobs run. Take the full sync duration from the replica's log: the time from the sync starting to MASTER <-> REPLICA sync: Finished with success. If a sync never finishes, estimate it as snapshot time, plus RDB size divided by network throughput, plus load time.

Then calculate:

• Output buffer limit: write rate x full sync duration, plus a margin. At 5 MB/s and a 10-minute sync, the buffer needs about 3 GB, so set 4 GB.
• Backlog: write rate x the longest disconnect you want to survive without a full sync. At 5 MB/s and 5 minutes, that's about 1.5 GB.

Keep the buffer's hard limit above repl-backlog-size.

#Stop the loop

Warning

Bigger buffers use real memory. Replica output buffers are not counted against maxmemory for eviction, so they can push the host into the OOM killer, on top of copy-on-write growth from the fork. Check free host memory against your calculated limit and rdb_last_cow_size before you raise anything.

Apply the changes on the primary. Both settings take effect without a restart.

redis-cli CONFIG SET client-output-buffer-limit "replica 4gb 2gb 300"
redis-cli CONFIG SET repl-backlog-size 1610612736

# Only if the logs show timeouts
redis-cli CONFIG SET repl-timeout 300

# Save to redis.conf so a restart keeps them
redis-cli CONFIG REWRITE

Apply the same settings on the replicas, because any of them can be promoted. repl-timeout has to be set on both sides, as each side enforces its own value. Sentinel and Cluster do not sync these settings between nodes.

Then lower the load while the sync catches up:

1. Pause batch jobs and bulk imports. This shrinks both the buffer growth and the copy-on-write memory.
2. Leave the replica alone. Restarting it starts the sync from zero.
3. Keep clients off the replica until master_link_status:up. It returns LOADING errors during the load.
4. Watch the buffer and the sync from the primary:

watch -n 10 "redis-cli CLIENT LIST TYPE replica; redis-cli INFO memory | grep -E 'mem_clients_slaves|mem_total_replication_buffers|mem_fragmentation_ratio'"

mem_total_replication_buffers exists on 7.0 and later only.

Caution

Don't fail over to a looping replica. Before any promotion, confirm master_link_status:up and a replica offset close to the primary's master_repl_offset. A replica that hasn't finished loading may hold old data or none at all, and promoting it loses writes.

#Shorten the sync itself

#ElastiCache and other managed services

You can't run CONFIG SET on most managed services. On Amazon ElastiCache:

• repl-backlog-size can be changed in the parameter group.
• client-output-buffer-limit-replica-hard-limit, -soft-limit and -soft-seconds are listed as not modifiable, so you can't apply the buffer fix above.
• reserved-memory-percent (default 25) sets aside memory for sync overhead. If nodes run tight during sync, keep it at the default or higher.

If replicas keep failing to sync on ElastiCache, reduce the write rate during the sync or scale up to a larger node type.

#Prevent it next time

• Alert when sync_full increases outside a planned change. A few per week is normal; several per hour is a loop.
• Alert on master_link_status:down on any replica for longer than your normal load time.
• Recalculate the buffer and backlog when the dataset grows or write rates change. Limits that were fine at 10 GB will fail at 50 GB.
• Test a full sync of a replica at peak write load before you rely on it for failover.
• Keep replica hosts the same size as the primary.