MongoDB multi-document transactions in production: a checklist for retries, WriteConflict, time limits, cache pressure and sharding

#Why a checklist

Transactions in MongoDB work, but they are a different cost model from single-document writes. The failures that reach on-call are predictable: transactions aborted after 60 seconds, retry storms on WriteConflict, commits whose result is unknown after a failover, and cache pressure from large transactions. Run through this list in design review and again before go-live.

#1. Do you need a transaction at all?

• A single-document update is atomic. If the data that must change together can live in one document (embedded array, sub-document), prefer that.
• Idempotent multi-step workflows (outbox pattern, compensating updates) may be cheaper than a transaction for long business processes.
• The transaction modifies a bounded number of documents. MongoDB's production guidance is to keep it to about 1,000 documents or fewer and break larger work into batches.

#2. Retry handling is in the driver's callback API

• Use the driver's callback API (session.withTransaction() in most drivers). It retries the whole transaction on errors labelled TransientTransactionError and retries the commit on UnknownTransactionCommitResult.
• The transaction body is safe to run more than once: no external side effects (emails, HTTP calls, message publishes) inside the callback.
• Code that uses the core API (startTransaction / commitTransaction) checks error labels, not just error codes:

// mongosh example with explicit retry handling
const session = db.getMongo().startSession();
const orders = session.getDatabase("shop").orders;
const stock  = session.getDatabase("shop").stock;

function runTxn() {
  session.startTransaction({ readConcern: { level: "snapshot" },
                             writeConcern: { w: "majority" } });
  try {
    stock.updateOne({ sku: "A1", qty: { $gte: 1 } }, { $inc: { qty: -1 } });
    orders.insertOne({ sku: "A1", qty: 1, ts: new Date() });
    commitWithRetry();
  } catch (e) {
    session.abortTransaction();
    throw e;
  }
}

function commitWithRetry() {
  while (true) {
    try { session.commitTransaction(); return; }
    catch (e) {
      if (e.hasErrorLabel && e.hasErrorLabel("UnknownTransactionCommitResult")) continue;
      throw e;
    }
  }
}

while (true) {
  try { runTxn(); break; }
  catch (e) {
    if (e.hasErrorLabel && e.hasErrorLabel("TransientTransactionError")) continue;
    throw e;
  }
}

• Retries are bounded and back off; an unbounded tight loop on a hot document turns contention into a CPU incident.

#3. Write conflicts are designed out

A WriteConflict (code 112) inside a transaction aborts it with TransientTransactionError. Unlike single-document writes, which retry internally, a transaction gives up and the driver retries from the start.

• No single "counter" or "status" document that every transaction updates. Shard counters across several documents or update them outside the transaction.
• Transactions touch documents in a consistent order to reduce conflicts.
• You measure aborts in load tests (section 8) and the abort rate is low at expected peak concurrency.

#4. Time and lock limits are understood

• The transaction finishes well under 60 seconds at p99, including network round trips from the app.
• You did not raise transactionLifetimeLimitSeconds to hide slow transactions. Long transactions pin WiredTiger history in cache and slow everything else.
• Every query inside the transaction is indexed. A collection scan inside a transaction holds a snapshot open for the whole scan.

Check the current values:

db.adminCommand({ getParameter: 1,
                  transactionLifetimeLimitSeconds: 1,
                  maxTransactionLockRequestTimeoutMillis: 1 });

#5. Read and write concerns match the guarantee you need

• Write concern is majority. A transaction committed with w: 1 can be rolled back after a failover, and the client was told it succeeded.
• Read concern is snapshot when the transaction must see a consistent point-in-time view across documents. (With majority write concern on commit, snapshot reads are guaranteed to come from majority-committed data.)
• Read preference inside the transaction is primary. Transactions with reads must use primary.

#6. Size and cache pressure

• Large transactions are split. All changes are held in the WiredTiger cache until commit; a few large transactions can push dirty cache up and stall other writes.
• You know how your version behaves under cache pressure. Recent versions abort transactions that are too large for the available cache with a TransactionTooLargeForCache error; the threshold is controlled by transactionTooLargeForCacheThreshold on versions that have it. Check your version's parameter reference before tuning it.
• Bulk loads and migrations run outside transactions, using ordered or unordered bulkWrite with idempotent upserts.

Watch the cache while load testing:

const c = db.serverStatus().wiredTiger.cache;
({ maxBytes: c["maximum bytes configured"],
   inCache:  c["bytes currently in the cache"],
   dirty:    c["tracked dirty bytes in the cache"] });

#7. Sharded clusters

• Transactions that touch one shard are much cheaper than cross-shard ones. Cross-shard transactions use two-phase commit coordinated by a shard and add round trips.
• The shard key is included in the filters of every operation inside the transaction, so mongos can target shards instead of broadcasting.
• Clients connect through mongos and you have tested a primary stepdown on a participant shard during a running transaction.
• Chunk migrations and the balancer are considered: migrations can cause transactions on moving chunks to abort with transient errors; the retry logic handles it.

#8. Observability and operations

Server-level counters:

const t = db.serverStatus().transactions;
({ active: t.currentActive, inactive: t.currentInactive, open: t.currentOpen,
   started: t.totalStarted, committed: t.totalCommitted, aborted: t.totalAborted });

• Alert on the abort ratio (totalAborted delta divided by totalStarted delta) above your baseline.
• Alert on currentOpen climbing without matching load; it means sessions are leaking open transactions.

Find long-running or idle open transactions:

db.getSiblingDB("admin").aggregate([
  { $currentOp: { allUsers: true, idleSessions: true } },
  { $match: { transaction: { $exists: true } } },
  { $project: { lsid: 1, client: 1, appName: 1, active: 1,
                "transaction.timeOpenMicros": 1,
                "transaction.parameters.txnNumber": 1 } },
  { $sort: { "transaction.timeOpenMicros": -1 } }
]);

Kill a stuck one by its session ID:

db.adminCommand({ killSessions: [ { id: UUID("<lsid.id from the output>") } ] });

Warning

Killing a session aborts its transaction; the application sees an error and, if it uses the callback API, may retry immediately. Find out why it stalled (an idle client holding an open transaction, a slow unindexed query) before killing repeatedly.

• Slow transactions are logged: the server logs transactions that exceed slowms with their duration and abort reason. Make sure those lines reach your log platform.
• The runbook for a failover includes checking for UnknownTransactionCommitResult spikes in application logs.

#9. Restrictions are checked in code review

• No writes to capped collections, and no reads or writes to config, admin or local databases inside a transaction.
• Collection and index creation inside transactions is only used where your version supports it, and not in hot paths.
• No count command inside transactions; use countDocuments() (an aggregation) instead.

#Prevent it next time

• Add the transaction checklist to design review templates for any service on MongoDB.
• Load test with realistic concurrency and a forced primary stepdown (rs.stepDown()) during the run; check abort and retry rates.
• Track p99 transaction duration and abort ratio on the same dashboard as WiredTiger dirty cache.
• Revisit data modelling when transactions grow: frequent large multi-document transactions often mean the document boundaries are wrong.