Home / Data Platform / Streaming & CDC

Data PlatformRunbookAdvanced

Sev1: Debezium connector failed — binlog/WAL position lost, how to recover without a full reload

Recover a Debezium CDC connector that fails because the source purged the binlog or WAL it needed, or its offsets were lost. Choose between incremental snapshots, a filtered re-snapshot or a new connector, and keep consumers consistent.

Deepak Bhardwaj3 min read

Applies toDebezium 2.x connectors for MySQL, PostgreSQL and SQL Server on Kafka Connect

Symptoms

  • MySQL: The connector is trying to read binlog starting at ... but this is no longer available on the server.
  • PostgreSQL: replication slot dropped or invalidated; replication slot "debezium" does not exist.
  • SQL Server: LSN no longer available in CDC tables (cleanup job removed it).
  • Connector task FAILED; downstream tables stop changing.
curl -s http://connect:8083/connectors/orders-cdc/status | jq

1. Why the position is gone

Engine Cause
MySQL binlog_expire_logs_seconds shorter than the connector outage
PostgreSQL Slot dropped manually or invalidated by max_slot_wal_keep_size
SQL Server CDC cleanup retention shorter than the outage
Any Connect offsets topic deleted or connector renamed (new name = new offsets)

2. Recovery options

Option Effect When
Incremental snapshot (signal-based) Re-reads tables in chunks while streaming continues Default choice on Debezium 1.6+/2.x
snapshot.mode=when_needed restart Full snapshot when offsets are invalid Small tables
New connector with a new name Fresh snapshot + streaming Offsets badly broken

Configure a signalling table (or Kafka signal topic) on the connector, then restart streaming from the current position and ask Debezium to re-snapshot the tables:

"signal.data.collection": "inventory.debezium_signal",
"snapshot.mode": "when_needed"

INSERT INTO inventory.debezium_signal (id, type, data)
VALUES ('resnap-20261025', 'execute-snapshot',
        '{"data-collections": ["inventory.orders", "inventory.order_lines"], "type": "incremental"}');

Options and the "when_needed"/"recovery" modes vary by connector and version — check the connector docs for exact names. Incremental snapshots emit READ (op: r) events for existing rows; streaming changes continue in between.

4. Consumers must handle re-delivered rows

A re-snapshot re-sends current rows. Sinks must upsert by primary key, not append. For a JDBC sink connector: insert.mode=upsert with pk.mode=record_key. For Spark/Delta sinks: MERGE on the key, with the source LSN/binlog position to ignore older versions.

Warning

Deletes that happened while the connector was down won't appear in a snapshot (the rows simply aren't there). Reconcile by comparing key sets between source and target, or rebuild the target table from the snapshot.

5. Restart and watch

curl -X POST http://connect:8083/connectors/orders-cdc/restart?includeTasks=true
curl -s http://connect:8083/connectors/orders-cdc/status | jq '.tasks[].state'

Watch connector metrics: MilliSecondsBehindSource, snapshot progress, Kafka topic throughput.

Prevent it next time

  • Source log retention longer than your worst-case connector outage (MySQL binlog days, SQL Server CDC retention).
  • PostgreSQL: monitor slot lag (pg_replication_slots), and set max_slot_wal_keep_size so a stuck slot can't fill the disk — knowing it may invalidate the slot.
  • Alert on connector task state and lag within minutes, not hours.
  • Configure the signalling table from day one so recovery is a SQL insert.

Related fixes