Sev1: Kafka consumer group lost its offsets — mass reprocessing or skipped data

#Symptoms

• Lag jumps to millions and downstream gets duplicates (reset to earliest).
• Lag drops to zero suddenly and data for a period never arrives (reset to latest).
• Happened after a consumer was down for a long time, a group was renamed, or someone ran a reset.

#1. Why were offsets lost?

Then auto.offset.reset decides where the consumer starts: earliest or latest.

kafka-consumer-groups.sh --bootstrap-server kafka1:9092 --describe --group orders-loader
kafka-consumer-groups.sh --bootstrap-server kafka1:9092 --describe --group orders-loader --members --verbose

#2. Stop the consumers

Offsets can only be reset when the group has no active members. Scale the consumer deployment to zero:

kubectl scale deploy/orders-loader --replicas=0 -n data

#3. Choose the right position

Best: last known good offsets (from your own checkpoint table, logs or a monitoring export of committed offsets). Next best: a timestamp just before the problem.

Dry run first:

kafka-consumer-groups.sh --bootstrap-server kafka1:9092 --group orders-loader --topic orders \
  --reset-offsets --to-datetime 2026-10-25T06:00:00.000 --dry-run

Execute:

kafka-consumer-groups.sh --bootstrap-server kafka1:9092 --group orders-loader --topic orders \
  --reset-offsets --to-datetime 2026-10-25T06:00:00.000 --execute

Other options: --to-offset, --shift-by -N, or --from-file offsets.csv (topic,partition,offset per line) for exact restore.

Warning

Resetting to a time re-delivers everything after it. Make sure the consumer writes idempotently (upsert/MERGE on a key) before you replay, or you trade missing data for duplicates.

#4. Data already deleted by retention

If the needed offsets are older than the topic's log start offset, Kafka no longer has the data. Recover from the source system (re-publish), from a sink (data lake copy) or from a MirrorMaker/replicated cluster.

#5. Restart and verify

kubectl scale deploy/orders-loader --replicas=6 -n data
kafka-consumer-groups.sh --bootstrap-server kafka1:9092 --describe --group orders-loader

Watch lag fall and reconcile counts downstream for the replayed window.

#Prevent it next time

• Raise offsets.retention.minutes above your longest plausible consumer outage.
• Pin group.id in config; review changes to it like a schema change.
• Periodically export committed offsets (a small job) so you always have a restore point.
• Idempotent sinks; alerts on sudden lag jumps and on groups with no members.