Sev1: MongoDB secondaries went too stale to sync. Oplog window exhausted and redundancy lost

#Symptoms

• rs.status() shows a secondary in RECOVERING with a message like too stale to catch up or error code RS102.
• Only the primary is healthy; a single failure now means downtime or data loss.
• w: "majority" writes slow or time out if a majority is no longer available.
• Usually after a long maintenance, a network outage, or a bulk load that churned the oplog fast.

#Step 1: confirm and measure (5 minutes)

rs.status().members.map(m => ({ name: m.name, state: m.stateStr, optime: m.optimeDate, msg: m.infoMessage || m.lastHeartbeatMessage }))
rs.printReplicationInfo()            // on the primary: oplog size and window ("log length start to end")
rs.printSecondaryReplicationInfo()   // lag per secondary

The oplog window is how much time the oplog covers at the current write rate. A secondary that is behind by more than the window cannot catch up, because the entries it needs are gone.

#Step 2: protect the primary

With redundancy lost:

• Postpone risky operations on the primary (index builds, upgrades, reboots).
• If a heavy batch job is churning the oplog, pause it; it makes the window shorter for the members you are about to resync.
• If writes use w: "majority" and only the primary plus arbiter remain, they may block. Do not lower write concern without the application owner's agreement.

#Step 3: grow the oplog first

Resyncing a large member takes time; if the oplog window is shorter than the resync, it fails again. Increase the oplog on the primary (online, MongoDB 4.0+):

use local
db.oplog.rs.stats().maxSize / 1024 / 1024 / 1024     // current max GB
db.adminCommand({ replSetResizeOplog: 1, size: 102400 })   // size in MB, e.g. 100 GB
// optional: keep at least N hours regardless of size (MongoDB 4.4+)
db.adminCommand({ replSetResizeOplog: 1, minRetentionHours: 48 })

Size it so the window covers at least the longest expected resync or maintenance plus margin. Check disk space on all members first; every member should get the same size.

Warning

minRetentionHours lets the oplog grow beyond its configured size to keep the hours you asked for. On a busy system that can use a lot of disk; monitor free space.

#Step 4: resync the stale secondary

#Option A: initial sync (simplest)

On the stale member:

sudo systemctl stop mongod
sudo mv /var/lib/mongodb /var/lib/mongodb.stale.$(date +%F)    # keep until done, if disk allows
sudo mkdir /var/lib/mongodb && sudo chown mongodb:mongodb /var/lib/mongodb
sudo systemctl start mongod

The member starts empty, joins the set and performs an initial sync (copies all data, then applies the oplog). Watch progress:

db.adminCommand({ replSetGetStatus: 1, initialSync: 1 }).initialSyncStatus

Initial sync of terabytes can take many hours and adds read load on its sync source. Point it at another secondary if one is healthy, or at the primary during low traffic:

db.adminCommand({ replSetSyncFrom: "db2.internal:27017" })   // run on the syncing member

#Option B: seed from a snapshot (faster for big data)

1. Take a filesystem or volume snapshot of a healthy member (consistent, with journaling on the same volume, or db.fsyncLock() while snapshotting).
2. Restore it onto the stale member's data path.
3. Start mongod; it catches up from the oplog, which must still cover the time since the snapshot.

This is why growing the oplog (step 3) comes first.

#Step 5: confirm

rs.status().members.map(m => [m.name, m.stateStr, m.health])
rs.printSecondaryReplicationInfo()

• All members PRIMARY/SECONDARY, lag near zero.
• Oplog window comfortably above your largest planned maintenance.
• Old data directories removed once the member is healthy.

#Why it happened: check the write pattern

// oplog growth rate: compare first and last entries
use local
const first = db.oplog.rs.find().sort({ $natural: 1 }).limit(1).next().ts
const last  = db.oplog.rs.find().sort({ $natural: -1 }).limit(1).next().ts
(last.t - first.t) / 3600     // hours in the window now

Common culprits: bulk updateMany jobs rewriting large documents, TTL deletes of huge volumes at once, or index builds plus loads. Spreading them out or batching with pauses keeps the window healthy.

#Prevent it next time

• Alert when the oplog window falls below a threshold (for example 24 hours) and when replication lag exceeds a few minutes.
• Size the oplog for the longest maintenance plus the time to rebuild the largest member.
• Plan maintenance on secondaries one at a time, and check the window before taking a member down.
• Use five-member replica sets (or three data-bearing members, no arbiter) for important data so one stale member does not leave you at a single copy.