Sev1: a production Kafka topic was deleted. Stop the damage, recreate it and recover the data

#Symptoms

• Producers log UNKNOWN_TOPIC_OR_PARTITION, or keep working because auto.create.topics.enable=true silently recreated the topic with default settings (often 1 partition, default retention).
• Consumers log offset-out-of-range and reset to earliest or latest depending on auto.offset.reset.
• Downstream systems see a gap, or a flood of reprocessed records.

#Why it hurts

Deleting a topic removes its log segments on every broker. There is no recycle bin in open-source Kafka. Recovery means getting the data from somewhere else: a mirror cluster, a sink, the original source, or backups.

#Step 1: stop it getting worse (5 minutes)

Check whether the topic was recreated by producers:

kafka-topics.sh --bootstrap-server $BS --describe --topic orders

If it exists with wrong settings (1 partition, wrong retention), producers are now writing to it. Decide quickly:

• If ordering by key matters, stop producers before more data lands in a topic with the wrong partition count.
• Disable auto-creation on the brokers to prevent repeats (dynamic config, applies cluster-wide):

kafka-configs.sh --bootstrap-server $BS --entity-type brokers --entity-default \
  --alter --add-config auto.create.topics.enable=false

If your Kafka version does not allow changing this dynamically, set it in server.properties and roll the brokers after the incident.

Pause consumers that would act on reprocessed or missing data (billing, notifications).

#Step 2: who and when

Broker logs and the controller log show the deletion:

grep -i "delet" /var/log/kafka/controller.log | grep orders | tail
grep -i "orders" /var/log/kafka/server.log | grep -i delet | tail

With authorizer logging enabled, kafka-authorizer.log shows the principal that performed DeleteTopics. Note the time: you need it to find replay start points.

#Step 3: recreate the topic correctly

Get the original configuration from your topic-as-code repo, a monitoring snapshot, or the mirror cluster:

kafka-topics.sh --bootstrap-server $MIRROR_BS --describe --topic primary.orders
kafka-configs.sh --bootstrap-server $MIRROR_BS --entity-type topics --entity-name primary.orders --describe

If producers auto-created a wrong topic, delete that (once producers are stopped) and recreate:

kafka-topics.sh --bootstrap-server $BS --delete --topic orders
kafka-topics.sh --bootstrap-server $BS --create --topic orders \
  --partitions 24 --replication-factor 3 \
  --config min.insync.replicas=2 --config retention.ms=604800000 --config cleanup.policy=delete

Warning

The partition count must match the original if consumers or downstream systems rely on key-to-partition mapping (for example compacted topics used as state, or Kafka Streams changelogs).

Recreate ACLs if they were topic-scoped (deleting the topic does not remove ACLs, but check):

kafka-acls.sh --bootstrap-server $BS --list --topic orders

#Step 4: recover the data

Pick the best source you have:

Example one-off replay with a simple consumer-producer (kcat):

kcat -C -b $MIRROR_BS -t primary.orders -o beginning -e -K: \
  | kcat -P -b $BS -t orders -K:

This preserves keys (and therefore partitioning, if partition counts match) but not original timestamps or headers by default. For production replays use a small program or MirrorMaker 2 so headers and timestamps survive.

#Step 5: fix consumer offsets

Consumer groups that committed offsets on the old topic have stale offsets (or none). Decide per group where it should restart:

kafka-consumer-groups.sh --bootstrap-server $BS --group billing --topic orders \
  --reset-offsets --to-datetime 2026-10-29T08:00:00.000 --dry-run
kafka-consumer-groups.sh --bootstrap-server $BS --group billing --topic orders \
  --reset-offsets --to-datetime 2026-10-29T08:00:00.000 --execute

Replayed messages get new timestamps unless the replay preserved them; in that case use --to-earliest and rely on idempotent consumers, or --to-offset per partition after checking the data.

Caution

Consumers that are not idempotent will double-process replayed records. Confirm deduplication (by event ID) before restarting billing-type consumers.

#Step 6: confirm

• Topic config matches the original; --describe shows full ISR.
• Producers running; consumer lag decreasing; no offset-out-of-range errors.
• Data owners confirm the recovered window (count by hour on source vs topic).

#Prevent it next time

• delete.topic.enable=false on production clusters, or restrict DeleteTopics with ACLs to a break-glass principal.
• auto.create.topics.enable=false; manage topics as code (Terraform, JulieOps, Strimzi KafkaTopic).
• Mirror critical topics to a second cluster or sink them to object storage.
• Require a ticket and a second person for any topic deletion in production.