Home / Data Platform / Streaming & CDC

Data PlatformRunbookAdvanced

Transformation: migrate a Kafka cluster from ZooKeeper to KRaft

Move a production Kafka cluster off ZooKeeper before ZooKeeper support disappears. Pre-checks, deploying KRaft controllers in migration mode, moving brokers, finalising, and the rollback window you get before finalisation.

Deepak Bhardwaj2 min read

Applies toApache Kafka 3.6–3.9 (ZooKeeper-to-KRaft migration), Confluent Platform 7.6+; Kafka 4.0+ is KRaft-only

Why now

Kafka 4.0 removed ZooKeeper mode. You must migrate on a 3.x release that supports migration (3.9 is the bridge release) before upgrading to 4.x. Check the exact supported path for your version in the Kafka upgrade notes.

1. Pre-checks

  • Brokers on a migration-capable version (upgrade first if not), with inter.broker.protocol.version at that version.
  • Note the cluster ID:
zookeeper-shell.sh zk1:2181 get /cluster/id
  • Check unsupported features for your version (some older features, like certain SCRAM or JBOD scenarios, had restrictions in early migration releases).
  • Back up ZooKeeper data and take a configuration snapshot of brokers.

2. Deploy KRaft controllers in migration mode

Provision 3 (or 5) controller nodes. Format their storage with the existing cluster ID:

kafka-storage.sh format -t <cluster-id> -c /etc/kafka/controller.properties

Controller config (key lines):

process.roles=controller
node.id=3000
controller.quorum.voters=3000@ctrl1:9093,3001@ctrl2:9093,3002@ctrl3:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093
zookeeper.metadata.migration.enable=true
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181

Newer releases use controller.quorum.bootstrap.servers and dynamic quorums; follow the documentation for your version.

Start controllers. The active controller waits for brokers to register before copying metadata from ZooKeeper.

3. Brokers: enable migration (rolling)

On each broker add:

zookeeper.metadata.migration.enable=true
controller.quorum.voters=3000@ctrl1:9093,3001@ctrl2:9093,3002@ctrl3:9093
controller.listener.names=CONTROLLER
listener.security.protocol.map=...,CONTROLLER:PLAINTEXT

Roll brokers one at a time, waiting for under-replicated partitions to return to zero between restarts. When all are restarted, the controller migrates metadata to KRaft (log: "Completed migration of metadata from ZooKeeper to KRaft"). The cluster is now in dual-write mode.

4. Brokers: move to KRaft mode (rolling)

Replace broker.id with node.id (same value), add process.roles=broker, and remove zookeeper.connect and the migration flag. Roll brokers again.

5. Finalise

Remove the migration settings from the controllers and restart them. ZooKeeper is no longer used.

Caution

Rollback to ZooKeeper is possible only before finalising (step 5), following the documented reverse steps. After finalisation, there's no way back — keep the cluster in the pre-finalised state for a soak period.

6. Verify

kafka-metadata-quorum.sh --bootstrap-server kafka1:9092 describe --status
kafka-topics.sh --bootstrap-server kafka1:9092 --describe --under-replicated-partitions

Then update monitoring (controller metrics move to the controller nodes), runbooks and tooling that called ZooKeeper directly (--zookeeper flags were removed long ago — scripts using them must use --bootstrap-server).

Checklist

  • On a migration-capable 3.x version; cluster ID recorded
  • Controllers formatted with the existing cluster ID
  • Brokers rolled with migration enabled; metadata migrated
  • Brokers rolled into KRaft mode; soak period completed
  • Finalised; ZooKeeper decommissioned; tooling updated

Related fixes