Home / Data Platform / Streaming & CDC

Data PlatformTroubleshootingAdvanced

Sev1: Kafka broker won't start — corrupt log segments, unclean shutdown or lost KRaft quorum

Get a Kafka broker or KRaft controller quorum back after a crash. Read the startup errors, handle long log recovery, corrupted index or segment files, wrong cluster IDs and lost controller quorum without making data loss worse.

Deepak Bhardwaj3 min read

Applies toApache Kafka 3.3+ (KRaft) and ZooKeeper-mode clusters, Confluent Platform

Read the startup log first

journalctl -u kafka -n 300 --no-pager
grep -E "ERROR|FATAL|Exception" /var/log/kafka/server.log | tail -50
Message contains Meaning Go to
Recovering unflushed segment, slow start Unclean shutdown; recovery in progress 1
CorruptRecordException, InvalidOffsetException, index errors Damaged segment or index 2
The Cluster ID ... doesn't match stored clusterId Wrong data dir or reformatted metadata 3
meta.properties missing / not formatted (KRaft) Storage not formatted for this node 3
Controller: no leader, RaftClient timeouts KRaft quorum lost 4
No space left on device Disk full Free space first

1. Slow recovery after an unclean shutdown

The broker checks segments that weren't flushed. With many partitions this can take a long time. Don't kill it — let it finish. Speed up next time with more threads:

num.recovery.threads.per.data.dir=8

2. Corrupted index or segment

Kafka rebuilds missing index files on start. For a corrupt index, stop the broker and move the index files (not the .log files) for the reported partition aside:

cd /var/lib/kafka/data/orders-7
mkdir -p /root/kafka-quarantine/orders-7 && mv 00000000000001234567.index 00000000000001234567.timeindex /root/kafka-quarantine/orders-7/

For a corrupt .log segment, the safest option on a replicated topic is to let the broker re-fetch the partition from the other replicas: stop the broker, move the whole partition directory aside, start the broker. It recreates the partition as a follower and copies data from the leader.

Caution

Only do that when other in-sync replicas exist for that partition (kafka-topics.sh --describe --topic orders). If this broker holds the only copy, you'd be deleting data.

3. Cluster ID / formatting problems

Usually caused by mounting the wrong disk, a new empty disk, or someone running kafka-storage.sh format again with a new cluster ID.

cat /var/lib/kafka/data/meta.properties
kafka-storage.sh info -c /etc/kafka/server.properties

Mount the correct volume. For a genuinely new empty disk on a KRaft broker, format it with the existing cluster ID:

kafka-storage.sh format -t <existing-cluster-id> -c /etc/kafka/server.properties --ignore-formatted

4. KRaft quorum lost

The controller quorum needs a majority (2 of 3, 3 of 5) to elect a leader.

kafka-metadata-quorum.sh --bootstrap-server kafka1:9092 describe --status
kafka-metadata-quorum.sh --bootstrap-server kafka1:9092 describe --replication

Bring back controllers with their original metadata log directories — that restores the majority. Rebuilding a controller from scratch with an empty metadata log when a majority is lost risks losing metadata; follow the Kafka documentation's recovery procedure for your version and get vendor support if you have it.

5. After it's up

kafka-topics.sh --bootstrap-server kafka1:9092 --describe --under-replicated-partitions
kafka-leader-election.sh --bootstrap-server kafka1:9092 --election-type preferred --all-topic-partitions

Prevent it next time

  • Controlled shutdown (systemctl stop with a long enough timeout) for maintenance.
  • Three or five controllers on separate failure domains; dedicated controller nodes for large clusters.
  • Separate disks per log directory with monitoring; replication factor 3 everywhere important.

Related fixes