Home / Data Platform / Streaming & CDC

Data PlatformRunbookAdvanced

Sev1: Kafka offline and under-replicated partitions after a broker loss

Restore a Kafka cluster where partitions are offline or under-replicated after one or more brokers failed. Find affected partitions, bring brokers back, decide on unclean leader election only as a last resort, and rebalance safely.

Deepak Bhardwaj2 min read

Applies toApache Kafka 3.x (ZooKeeper and KRaft), Confluent Platform, kafka-topics.sh / kafka-metadata tools

Symptoms

  • Producers fail with NotLeaderOrFollowerException or NOT_ENOUGH_REPLICAS; consumers stall on some partitions.
  • OfflinePartitionsCount > 0 on the controller; UnderReplicatedPartitions > 0 on brokers.

1. Which partitions?

kafka-topics.sh --bootstrap-server kafka1:9092 --describe --unavailable-partitions
kafka-topics.sh --bootstrap-server kafka1:9092 --describe --under-replicated-partitions
kafka-topics.sh --bootstrap-server kafka1:9092 --describe --under-min-isr-partitions
State Impact
Under-replicated, ISR ≥ min.insync.replicas Still serving; reduced durability
Under min ISR Producers with acks=all fail; reads work
Offline (no leader) Partition unavailable for reads and writes

2. Bring the brokers back

Almost always the right fix. Check why they died:

journalctl -u kafka --since "1 hour ago" | tail -100
df -h /var/lib/kafka          # full log disk is the top cause
grep -E "OutOfMemory|FATAL|Shutdown" /var/log/kafka/server.log | tail
  • Disk full: free space (delete old non-Kafka files, extend the volume), never delete partition segment files by hand while the broker runs.
  • OOM: check heap settings and page cache pressure.
  • Hardware/VM loss: start a replacement with the same broker.id / node.id and the same log directories (reattach disks) so it rejoins as the same replica.

As brokers return, followers catch up and ISR recovers. Watch:

watch -n 10 "kafka-topics.sh --bootstrap-server kafka1:9092 --describe --under-replicated-partitions | wc -l"

3. Last resort: unclean leader election

If the only in-sync replica is permanently lost, the partition stays offline. You can allow an out-of-sync replica to become leader:

kafka-leader-election.sh --bootstrap-server kafka1:9092 --election-type unclean --topic orders --partition 7

Caution

Unclean election loses messages that were only on the failed leader. Get sign-off from the data owner, record the offsets, and plan reconciliation from the source system.

4. Restore balance

After recovery, leadership may be concentrated on a few brokers:

kafka-leader-election.sh --bootstrap-server kafka1:9092 --election-type preferred --all-topic-partitions

If a broker is permanently gone, reassign its replicas to the remaining/new brokers with kafka-reassign-partitions.sh (generate, review, execute with --throttle so replication doesn't saturate the network).

kafka-reassign-partitions.sh --bootstrap-server kafka1:9092 --reassignment-json-file move.json --execute --throttle 50000000
kafka-reassign-partitions.sh --bootstrap-server kafka1:9092 --reassignment-json-file move.json --verify

Prevent it next time

  • Replication factor 3 and min.insync.replicas=2 for important topics; producers with acks=all.
  • unclean.leader.election.enable=false (default) so data loss is an explicit decision.
  • Rack awareness (broker.rack) across zones; alerts on UnderReplicatedPartitions, OfflinePartitionsCount and disk usage at 75%.

Related fixes