Kafka partition reassignment in production: rack awareness, throttling and rebalancing brokers safely

Adding a broker does not move any data onto it. Kafka keeps existing partitions where they are until you reassign them. Reassignment copies whole partitions over the network, which can saturate disks and links and slow the whole cluster if you do it carelessly.

#When you need it

• New brokers added (they are empty).
• A broker is being decommissioned.
• Disk usage is uneven across brokers.
• Replicas are not spread across racks / availability zones.

#Check the current layout

kafka-topics.sh --bootstrap-server $BS --describe --topic orders
kafka-log-dirs.sh --bootstrap-server $BS --describe --broker-list 1,2,3,4 \
  | grep -o '"size":[0-9]*' | head
kafka-broker-api-versions.sh --bootstrap-server $BS | grep -o 'id: [0-9]* rack: [^)]*'

Each broker should have broker.rack set (for example the AZ name). With racks set, Kafka places replicas of a partition on different racks when creating topics and generating plans.

#Step 1: generate a plan

cat > topics.json <<'EOF'
{"version":1,"topics":[{"topic":"orders"},{"topic":"payments"}]}
EOF

kafka-reassign-partitions.sh --bootstrap-server $BS \
  --topics-to-move-json-file topics.json --broker-list "1,2,3,4,5,6" --generate \
  > plan-raw.txt

The output contains the current assignment (save it: that is your rollback) and a proposed one. Split them into rollback.json and plan.json.

The generated plan reshuffles almost everything, which moves far more data than needed. For big clusters, write a minimal plan by hand or with a script that only moves enough partitions to the new brokers, or use Cruise Control, which optimises for minimal movement and many goals (disk, network, rack, leader balance).

Warning

Never change the first replica of many partitions at once without planning leader elections. The first replica is the preferred leader; moving it shifts client traffic to new brokers.

#Step 2: execute with a throttle

kafka-reassign-partitions.sh --bootstrap-server $BS \
  --reassignment-json-file plan.json --execute --throttle 50000000    # 50 MB/s per broker

The throttle limits replication traffic for the moving replicas only. Start conservative; raise it if producers and consumers are unaffected:

kafka-reassign-partitions.sh --bootstrap-server $BS \
  --reassignment-json-file plan.json --execute --additional --throttle 100000000

Move in batches (for example 20-50 partitions or a few hundred GB at a time) rather than the whole cluster at once.

#Step 3: monitor

kafka-reassign-partitions.sh --bootstrap-server $BS --reassignment-json-file plan.json --verify
kafka-topics.sh --bootstrap-server $BS --describe --under-replicated-partitions

Watch at the same time:

--verify also removes the throttle once the reassignment is complete. Always run it at the end; a leftover throttle silently slows normal replication later.

#Step 4: rebalance leaders

After movement, run a preferred leader election so leadership matches the plan:

kafka-leader-election.sh --bootstrap-server $BS --election-type PREFERRED --all-topic-partitions

auto.leader.rebalance.enable=true (default) does this periodically, but running it right away gets traffic onto the new brokers sooner.

#Decommissioning a broker

1. Generate a plan with the broker excluded from --broker-list.
2. Execute with throttle, verify until complete.
3. Confirm nothing is left on it:

kafka-log-dirs.sh --bootstrap-server $BS --describe --broker-list 4 | grep -c '"partition"'

4. In KRaft mode, after shutting it down, unregister it: kafka-cluster.sh unregister --bootstrap-server $BS --id 4 (check your version's tooling).

#Stuck or harmful reassignments

kafka-reassign-partitions.sh --bootstrap-server $BS --reassignment-json-file plan.json --cancel

#Moving between disks on one broker

JBOD setups can move a replica between log dirs on the same broker by specifying log_dirs in the plan. Inter-disk moves are throttled with --replica-alter-log-dirs-throttle.

#Prevent it next time

• Set broker.rack on every broker from day one.
• Keep disk usage below ~70% so you have room to move data.
• Automate balancing with Cruise Control (or your managed service's rebalancing) and review its proposals.
• Keep the rollback JSON for every reassignment in your change record.