MongoDB change stream fails with ChangeStreamHistoryLost: resume tokens, the oplog window and large events

#The symptom

A change stream consumer (a custom service, the MongoDB Kafka connector, Debezium, a search indexer) restarts and fails straight away:

MongoServerError: Error on remote shard ... :: caused by ::
Resume of change stream was not possible, as the resume point may no longer be in the oplog.
code: 286, codeName: 'ChangeStreamHistoryLost'

Related failures you will see on the same pipelines:

The first one is the common one. The consumer was down, paused or too slow for longer than the oplog window.

#How resume really works

Each event carries an _id that is the resume token. It encodes the cluster time of the oplog entry. When you pass it back in resumeAfter or startAfter, every node serving the stream must still have that oplog entry. If the oldest entry in the oplog is newer than the token, the server refuses with 286 rather than silently skipping events.

Three details catch experienced teams out:

• The window is the smallest one in the replica set. The stream can be served by any member your read preference picks. A secondary with a smaller oplog, or one that was just resynced, has a shorter window.
• On sharded clusters every shard must cover the token. One busy shard with a short window breaks the stream for the whole cluster.
• Quiet collections age out tokens. If you watch one collection that rarely changes and only save the token when an event arrives, the saved token gets older while the oplog keeps cycling from other writes. Every event, including post-batch tokens with no events, should advance the saved token.

#Checks

#What is the oplog window on each member?

// Run on each member, or on each shard's primary and secondaries
rs.printReplicationInfo()
// configured oplog size:   51200MB
// log length start to end: 20412secs (5.67hrs)

db.getSiblingDB("local").oplog.rs.find({}, {ts: 1}).sort({$natural: 1}).limit(1)

log length start to end is the window. Compare it with the longest time your consumer has ever been down, plus the time to replay its backlog.

#How old is the consumer's saved token?

Decode the timestamp the consumer stored. In a running stream the post-batch token is available from the driver:

from pymongo import MongoClient

client = MongoClient("mongodb://cdc-user@rs0-a,rs0-b,rs0-c/?replicaSet=rs0")
coll = client.shop.orders

with coll.watch(full_document="updateLookup") as stream:
    for change in stream:
        handle(change)
        save_token(stream.resume_token)   # advances even when filters drop events

Compare clusterTime of the last processed event with the first ts in the oplog. If the gap is close to the window, you are one outage away from 286.

#Is the oplog being churned by something unexpected?

A bulk job, a $merge into a large collection or a TTL index deleting millions of documents can shrink a 24-hour window to an hour. Check the oplog write rate by namespace over a recent slice:

var since = Timestamp(Math.floor(Date.now()/1000) - 600, 0);
db.getSiblingDB("local").oplog.rs.aggregate([
  { $match: { ts: { $gte: since } } },
  { $group: { _id: { ns: "$ns", op: "$op" }, n: { $sum: 1 } } },
  { $sort: { n: -1 } }, { $limit: 10 }
])

This scans the last ten minutes of the oplog. Run it on a secondary.

#Fix: re-sync the consumer without losing events

There is no way to resume from a token that is no longer in the oplog. You have to rebuild the consumer's view from a snapshot and then stream from a known point.

1. Record the current cluster time before you take the snapshot:

db.adminCommand({ hello: 1 }).operationTime
// Timestamp({ t: 1762700000, i: 1 })

2. Run the full load of the target (initial sync in Kafka connector or Debezium terms, or your own export).
3. Start the stream with startAtOperationTime set to the time from step 1. Events applied twice must be idempotent (upserts keyed on _id), because the snapshot and the stream overlap.

from bson.timestamp import Timestamp
stream = coll.watch(start_at_operation_time=Timestamp(1762700000, 1),
                    full_document="updateLookup")

For the Kafka connector the documented recovery is to reset its stored offsets (which hold the resume token) and let it copy existing data again. For Debezium, use a new snapshot or an incremental snapshot signal.

Warning

Clearing the saved token and starting "from now" makes the error go away but drops every change made while the consumer was down. Only do that if downstream can tolerate the gap, and write down the window you skipped.

#Other failure modes on the same pipeline

#Events larger than 16 MB

fullDocument: 'updateLookup' plus a pre-image can push an event over the BSON limit. From MongoDB 7.0 (and 6.0.9+) add $changeStreamSplitLargeEvent as the last stage; the consumer must reassemble fragments.

db.orders.watch([{ $changeStreamSplitLargeEvent: {} }], { fullDocument: "updateLookup" })

Better still, project only the fields the consumer needs, so events stay small.

#updateLookup returns the wrong document

updateLookup reads the current document when the event is delivered, not the version at the time of the update. A lagging consumer sees later state, or null if the document was deleted. If you need the exact post-image, enable pre- and post-images on the collection and use fullDocument: 'whenAvailable' or 'required':

db.runCommand({ collMod: "orders", changeStreamPreAndPostImages: { enabled: true } })

// Pre- and post-images are stored in a system collection; cap their age
db.adminCommand({ setClusterParameter: {
  changeStreamOptions: { preAndPostImages: { expireAfterSeconds: 86400 } } } })

Images add write volume and storage. Set the expiry to at least your oplog window, or required lookups fail with a missing image.

#Sharded cluster stream stalls

mongos merges events from every shard in cluster-time order, so it waits for the slowest shard. A shard with no writes still sends periodic no-op entries, but a shard that is overloaded or has high replication lag holds up the whole stream. Check rs.printSecondaryReplicationInfo() on each shard.

#resumeAfter fails after a collection drop or rename

The stream returns an invalidate event. resumeAfter can't move past it; startAfter with the invalidate event's token can.

#Prevent it next time

Size the oplog from what the consumer needs, not the default. Set a minimum retention so the window doesn't shrink below your target during write bursts (MongoDB 4.4+). Size in MB and retention in hours:

db.adminCommand({ replSetResizeOplog: 1, size: 102400, minRetentionHours: 48 })

Run it on every member, including every shard. With minRetentionHours the oplog can grow past size during bursts, so leave disk headroom. On Atlas, set the minimum oplog window in the cluster's additional settings.

• Alert on the oplog window per member, for example below twice the consumer's maximum tolerated downtime.
• Alert on consumer lag in seconds (current cluster time minus last processed clusterTime), not in event counts.
• Save the post-batch resume token on every batch, even an empty one. For the Kafka connector, enable heartbeats for quiet namespaces.
• Make consumers idempotent so a snapshot-plus-replay re-sync is always safe.
• Put a re-sync runbook in place before you need it: who triggers the snapshot, how long it takes, and what downstream sees meanwhile.