MongoDB writes disappeared after a failover: rollback files, write concern and recovery

#Symptom

After a failover or network blip, an application reports that records it saved have gone. The driver returned success. There are no errors in the app, and the cluster looks healthy again.

This is usually a replica set rollback, not corruption.

#How it happens

1. The primary accepts writes that haven't yet reached a majority of members.
2. It's cut off or crashes, and a secondary is elected as the new primary.
3. The old primary returns, and finds that its oplog has diverged.
4. It undoes the writes the new primary doesn't have, and saves them to rollback files.

If those writes were acknowledged with w: 1, the application thinks they're safe. Writes acknowledged by a majority, with journaling, aren't rolled back in this way.

#Step 1: don't restart or resync yet

The rollback files are on the old primary's disk. Don't resync or rebuild this member, or wipe its data, until you've saved them. If it's a VM or container, check that its volume is persistent.

#Step 2: confirm a rollback happened

Look in the log of the member that was primary before. On modern versions, logs are JSON:

grep '"c":"ROLLBACK"' /var/log/mongodb/mongod.log | head -30

Then check the current replica set state:

rs.status().members.map(m => ({
  name: m.name,
  state: m.stateStr,
  lastHB: m.lastHeartbeat,
  optime: m.optimeDate,
  uptime: m.uptime
}))

rs.printReplicationInfo()
rs.printSecondaryReplicationInfo()

Exact log messages vary across versions. Rollback lines record the common point and the time span. From this, work out the window of affected writes.

#Step 3: find the rollback files

They're kept under the data directory, per collection:

DBPATH=$(mongosh --quiet --eval 'db.serverCmdLineOpts().parsed.storage.dbPath')
find "$DBPATH/rollback" -type f -name '*.bson' -exec ls -lh {} \;

The structure generally looks like this:

<dbPath>/rollback/<collection-uuid>/removed.<timestamp>.bson

The directory uses the collection's UUID on recent versions. To map a UUID back to a name:

db.getSiblingDB("shop").getCollectionInfos().map(c => ({ name: c.name, uuid: c.info.uuid }))

If there's no rollback directory, check whether createRollbackDataFiles was set to false:

db.adminCommand({ getParameter: 1, createRollbackDataFiles: 1 })

First, copy them somewhere safe, off this server:

cp -a "$DBPATH/rollback" /backup/rollback-$(date +%F-%H%M)

#Step 4: read what was rolled back

The files are BSON, so convert them to look at them:

bsondump /backup/rollback-2026-11-07-1012/<uuid>/removed.<timestamp>.bson > rolled-back.json

wc -l rolled-back.json
head -5 rolled-back.json

Check each document against the current data, because a user or app may have already re-created it after the failover:

const rolled = ["<id1>", "<id2>"];   // _id values from the file
db.getSiblingDB("shop").orders.find({ _id: { $in: rolled.map(id => ObjectId(id)) } })

The file contains the document as it was just before rollback. It isn't a list of operations, so rolled-back deletes need special care.

#Step 5: restore what's safe

Have the data owner approve this. The safest way is to load into a separate collection, and then merge only what's agreed on:

mongorestore --host primary01:27017 \
  --db recovery --collection orders_rollback_20261107 \
  /backup/rollback-2026-11-07-1012/<uuid>/removed.<timestamp>.bson

Then count how many are truly missing from the live collection:

const ids = db.getSiblingDB("recovery").orders_rollback_20261107.distinct("_id");
const found = db.getSiblingDB("shop").orders.countDocuments({ _id: { $in: ids } });
print(`rolled back: ${ids.length}, still present: ${found}, missing: ${ids.length - found}`);

Once the owner has agreed, insert just the missing ones with $merge, which can write to another database and leaves existing documents untouched:

db.getSiblingDB("recovery").orders_rollback_20261107.aggregate([
  { $merge: {
      into: { db: "shop", coll: "orders" },
      on: "_id",
      whenMatched: "keepExisting",
      whenNotMatched: "insert"
  } }
])

Caution

Never restore directly into the live collection without review. You might bring back documents that were correctly removed or superseded after the failover. Test with a count first, and keep your copy of the rollback files until the business signs off.

#Step 6: find out why writes weren't durable

Check the cluster-wide default:

db.adminCommand({ getDefaultRWConcern: 1 })

Since 5.0, the implicit default is usually w: "majority", unless your topology has arbiters that prevent it. If apps show a lost write, one of these usually applies:

Check how the arbiter affects you:

rs.conf().members.map(m => ({ host: m.host, arbiter: m.arbiterOnly, votes: m.votes, priority: m.priority }))

Set a clear default for the cluster:

db.adminCommand({
  setDefaultRWConcern: 1,
  defaultWriteConcern: { w: "majority", wtimeout: 5000 },
  defaultReadConcern: { level: "local" }
})

#Step 7: use the right concern in the app

mongodb://db1,db2,db3/shop?replicaSet=rs0&w=majority&journal=true&retryWrites=true&wtimeoutMS=5000

A wtimeout error doesn't mean that the write failed. It may still succeed, so the app needs to handle "unknown". Make operations safe to retry.

Warning

Majority writes cost some latency, because each one waits for a replica. Measure it, but don't trade durability for speed on data you can't lose. If a majority is slow, look at the replication lag and the network, rather than lowering the write concern.

Reads also matter. A dirty read with readConcern: "local" from the primary can return data that's later rolled back. So a user might see a record that disappears. Use majority when that would be harmful.

#Test it

On a non-production replica set:

1. Stop replication to the secondaries, or use a firewall to isolate the primary.
2. Write with w: 1, then step down the primary, and bring it back.
3. Look for rollback files.
4. Repeat with w: "majority", and confirm that the write either times out or survives.

#Prevent it next time

• Set w: "majority" as the default, and search your code for lower values.
• Use three data-bearing members in place of a PSA set.
• Alert on replica set elections and rollback log lines, and monitor free disk for rollback files.
• Keep the mongosh cheat sheet and your backups ready. If there's no primary, see no primary.