In KRaft mode the controller quorum is the cluster's metadata store. If it loses its leader, or a voter falls far behind, brokers cannot change topics, ISR state or leadership. This sheet covers the commands you need when you are checking or changing the quorum. It assumes $BS is a broker bootstrap address and $BC is a controller address, for example ctl1:9093.
Caution
A quorum of N voters needs a majority alive to elect a leader and commit metadata. Three voters tolerate one failure and five tolerate two. Never take down or remove a voter while another voter is already unhealthy.
Quorum health
Leader, epoch and follower lag
# Through the brokers
kafka-metadata-quorum.sh --bootstrap-server $BS describe --status
# Directly against the controllers (works when brokers are down; 3.7+)
kafka-metadata-quorum.sh --bootstrap-controller $BC describe --status
The fields that matter:
| Field | Healthy | Worry when |
|---|---|---|
LeaderId | A controller node ID | -1 or changing on every call |
LeaderEpoch | Stable | Climbing quickly, which means repeated elections |
HighWatermark | Increasing | Flat while the cluster is changing |
MaxFollowerLag | Small, usually tens of records | Thousands and growing |
MaxFollowerLagTimeMs | Low milliseconds to a few seconds | Seconds and growing |
CurrentVoters | All expected controllers | One is missing, or the IDs are wrong |
CurrentObservers | Every broker | A broker is missing, so it is not fetching metadata |
Per-replica replication
kafka-metadata-quorum.sh --bootstrap-controller $BC describe --replication
This prints one row per voter and observer with NodeId, DirectoryId (3.9+), LogEndOffset, Lag, LastFetchTimestamp, LastCaughtUpTimestamp and Status. If a voter's LastCaughtUpTimestamp is minutes old, that controller is effectively out of the quorum even though its process is up. Check its disk latency, GC pauses and network path to the leader.
JMX metrics worth alerting on
Metric names vary by version, so check your build's MBean tree.
| MBean / metric | Alert when |
|---|---|
kafka.controller:type=KafkaController,name=ActiveControllerCount | Sum across controllers is not exactly 1 |
kafka.server:type=raft-metrics current-leader, current-epoch | The leader is -1, or the epoch keeps rising |
kafka.server:type=raft-metrics commit-latency-avg | Sustained increase, usually from disk fsync latency |
kafka.controller:type=KafkaController,name=LastAppliedRecordLagMs | Grows, which means the controller is falling behind its own log |
kafka.controller:type=ControllerEventManager,name=EventQueueTimeMs | High p99, which means the controller event loop is saturated |
kafka.controller:type=KafkaController,name=MetadataErrorCount | Any non-zero value |
Feature and metadata versions
kafka-features.sh --bootstrap-server $BS describe
kafka-features.sh --bootstrap-controller $BC describe
The output lists metadata.version and kraft.version, plus other finalized features. If kraft.version is 0 or missing, the quorum is static: it is defined by controller.quorum.voters. If it is 1, the quorum is dynamic: it uses controller.quorum.bootstrap.servers, and you can change voters at runtime.
# Preview, then finalize the metadata version after all nodes run the new binaries
kafka-features.sh --bootstrap-server $BS upgrade --release-version 4.1 --dry-run
kafka-features.sh --bootstrap-server $BS upgrade --release-version 4.1
# Older 3.x tools use --metadata instead of --release-version
kafka-features.sh --bootstrap-server $BS upgrade --metadata 3.9
# 4.1+: convert a static quorum to a dynamic one
kafka-features.sh --bootstrap-server $BS upgrade --feature kraft.version=1
Warning
Finalizing a
metadata.versioncan make downgrade impossible, depending on which versions you cross. Bake the new binaries first and finalize last. Afterkraft.version=1, roll the nodes to replacecontroller.quorum.voterswithcontroller.quorum.bootstrap.servers. A dynamic quorum must not havecontroller.quorum.votersset. Follow the KRaft operations page for your exact release.
Formatting controllers
# One cluster ID for the whole cluster
kafka-storage.sh random-uuid
# First controller of a new dynamic quorum
kafka-storage.sh format --cluster-id $CID --config controller.properties --standalone
# Or bootstrap all initial voters at once (id@host:port:directory-id)
kafka-storage.sh format --cluster-id $CID --config controller.properties \
--initial-controllers "1@ctl1:9093:<dir1>,2@ctl2:9093:<dir2>,3@ctl3:9093:<dir3>"
# A controller that will be added to an existing dynamic quorum
kafka-storage.sh format --cluster-id $CID --config controller.properties --no-initial-controllers
Check the node ID and directory ID that formatting wrote:
cat /var/lib/kafka/meta/meta.properties # node.id, directory.id, cluster.id
Adding and removing controllers (dynamic quorum)
Add a voter
Start the newly formatted controller. It joins as an observer and catches up. Then promote it to a voter. Run this on the new controller's host, because the tool reads that node's config and meta.properties:
kafka-metadata-quorum.sh --command-config /etc/kafka/controller.properties \
--bootstrap-controller ctl1:9093 add-controller
Confirm that it appears in CurrentVoters and that its Lag in describe --replication stays near zero.
Remove a voter
kafka-metadata-quorum.sh --bootstrap-controller ctl1:9093 remove-controller \
--controller-id 3 --controller-directory-id <directory.id of node 3>
Newer releases can work out the directory ID from the controller ID, so the flag is optional there. Pass it anyway, because it guards against removing the wrong incarnation of a node.
Replacing a failed controller disk
Treat a wiped disk as a new voter. Format it with --no-initial-controllers, start it, remove the old directory ID with remove-controller, then run add-controller. Do this one voter at a time, and wait for lag to reach zero before you touch the next one.
Tip
Keep the quorum size odd. To move from 3 to 5 voters, add two voters one after the other. To replace a host, add the new voter first and remove the old one second, so you never drop below the original fault tolerance.
Reading metadata offline
Metadata shell on a snapshot
ls /var/lib/kafka/meta/__cluster_metadata-0/
kafka-metadata-shell.sh --snapshot \
/var/lib/kafka/meta/__cluster_metadata-0/00000000000000123456-0000000042.checkpoint
Inside the shell, use ls, cd, cat, tree and find, for example ls /image/topics/byName and cat /image/cluster/.... The node tree layout varies by version. Run it against a copy of the directory, or on a stopped node.
Decoding the metadata log
kafka-dump-log.sh --cluster-metadata-decoder \
--files /var/lib/kafka/meta/__cluster_metadata-0/00000000000000000000.log | tail -50
Use this to see the last records a controller applied before it crashed, such as a PartitionChangeRecord storm or a fencing loop.
Symptom to first command
| Symptom | First command | Likely cause |
|---|---|---|
| Topic creates time out | describe --status | No leader, or a voter majority is down |
| Leader epoch climbing | describe --replication | A voter with slow disk or network keeps timing out |
| One broker never gets new topics | Check CurrentObservers | The broker is not fetching metadata (listener, auth, controller.listener.names) |
add-controller rejected | kafka-features.sh describe | The quorum is static (kraft.version 0) |
| Controller won't start after disk loss | cat meta.properties | Directory ID mismatch; reformat it and add it as a new voter |
Prevent it next time
- Put controllers on dedicated, low-latency disks. Metadata commits fsync, so a noisy disk shows up as elections.
- Spread voters across racks or availability zones. Three AZs and three voters is the usual minimum.
- Alert on
ActiveControllerCountnot equal to 1, on epoch changes, and on voter lag. Brokers continue serving data from cached metadata, so a sick quorum can stay invisible until the next change. - Record each voter's node ID and directory ID in your runbook, so a removal never needs guesswork.