Home / Databases / MongoDB

DatabasesRunbookAdvanced

Sev1: MongoDB replica set has no primary and writes fail

Applications get NotWritablePrimary or "no primary found", and rs.status() shows only SECONDARY members. Work out whether a voting majority is reachable, fix network or member problems, and use a forced reconfiguration only when the majority is truly gone.

Deepak Bhardwaj3 min read

Applies toMongoDB 6.0, 7.0, 8.0 replica sets (self-managed); sharded cluster shards behave the same per shard

Symptom

MongoServerSelectionError: Server selection timed out ... ReplicaSetNoPrimary
NotWritablePrimary

rs.status().members.map(m => ({ name: m.name, state: m.stateStr, health: m.health, lastHeartbeatMessage: m.lastHeartbeatMessage }))

All reachable members are SECONDARY (or RECOVERING), none PRIMARY.

Why there's no primary

A primary needs votes from a majority of voting members. In a 3-member set, 2 must see each other.

Cause Typical sign
Majority of members down health: 0 for 2 of 3 members
Network partition Members healthy but can't reach each other; heartbeat timeouts
Remaining members not electable Priority 0, hidden, arbiter only, or too far behind (catchUpTimeoutMillis)
Clock / DNS / TLS problems after a change Heartbeat errors mentioning hostnames or certificates
Members in RECOVERING / STARTUP2 Initial sync or too stale to recover

Step 1: count voting members you can reach

rs.conf().members.map(m => ({ host: m.host, votes: m.votes, priority: m.priority, arbiterOnly: m.arbiterOnly }))

From each node, test connectivity to the others on port 27017 (nc -zv host 27017) and check that the hostnames in rs.conf() resolve correctly from each node.

Step 2: fix the normal causes first

  • Start stopped mongod processes; check their logs (/var/log/mongodb/mongod.log) for startup errors such as disk full or a lock file.
  • Fix network/security group/firewall rules between members.
  • Fix DNS (members are addressed by the hostnames in the config).
  • Renew expired TLS certificates used for member authentication.

Once a majority can talk, an election happens within seconds (electionTimeoutMillis, default 10 s).

Step 3: majority is gone for good

Example: 3-member set in two data centres, DC1 (2 members) burned down, DC2 has 1 member. No election is possible. To restore writes, force a new configuration on the surviving member:

cfg = rs.conf()
cfg.members = cfg.members.filter(m => m.host === "mongo3.dc2.example.com:27017")
rs.reconfig(cfg, { force: true })

The surviving member becomes primary of a one-member set. Then add new members as soon as possible:

rs.add({ host: "mongo4.dc2.example.com:27017" })
rs.add({ host: "mongo5.dc2.example.com:27017" })

Caution

A forced reconfig on a member that's behind loses writes it never received. If the "lost" members come back with their old config, they must not rejoin as-is: they may have writes the new primary doesn't (rolled back into rollback/ files) and can cause a split-brain if the network was only partitioned. Make sure the old members are really down, then wipe and resync them.

Step 4: after a primary exists

  • rs.status(): one PRIMARY, secondaries catching up (optimeDate).
  • Look for rollback files in each member's dbpath/rollback/ and review them with the application team.
  • Check write concern: apps using w: "majority" will wait or fail until enough members are back.

Design lessons

Problem Fix
2 members + arbiter in one site Spread voting members across 3 failure domains (e.g. 3 zones or 2 sites + tie-breaker)
Only one DC survives in practice Plan for a forced reconfig with a written runbook, or use 3 sites
Arbiter with w: majority Avoid arbiters when using majority write concern; they count for elections but hold no data

Prevent it next time

  • Odd number of voting members across independent failure domains.
  • Monitoring on member state and replication lag, not just process health.
  • DR runbook for forced reconfig, rehearsed.

Related fixes