Kafka rolling restart runbook: restart brokers and controllers one at a time without offline partitions or client errors

A rolling restart for a config change, JVM patch, OS patch or Kafka upgrade should be invisible to clients. When it isn't, the usual cause is one of these: someone restarted the next broker before the previous one had caught up, a topic had replication factor 1, or slow log recovery stretched a 30-second restart into 20 minutes. This runbook adds a gate between every node.

#Before you start

#Pre-flight checks

All of these must come back empty, with every broker registered:

BS=kafka1:9092
kafka-topics.sh --bootstrap-server $BS --describe --under-replicated-partitions
kafka-topics.sh --bootstrap-server $BS --describe --under-min-isr-partitions
kafka-topics.sh --bootstrap-server $BS --describe --unavailable-partitions
kafka-metadata-quorum.sh --bootstrap-server $BS describe --status
kafka-broker-api-versions.sh --bootstrap-server $BS | grep -c '^[^ ].*id:'

Next, find partitions that will go offline, or reject acks=all writes, when any single broker stops:

# Replication factor 1: these go offline while their broker restarts
kafka-topics.sh --bootstrap-server $BS --describe \
  | awk '/ReplicationFactor: 1/ {print $2}'

# Already at min ISR: one more stop makes producers fail with NOT_ENOUGH_REPLICAS
kafka-topics.sh --bootstrap-server $BS --describe --at-min-isr-partitions

Warning

A topic with RF=3 and min.insync.replicas=3 cannot take acks=all writes while any one of its replicas is down. Either agree on that outage with the owners, or lower the min ISR for the duration of the change and record that you did. RF=1 topics are unavailable while their broker restarts, whatever you do.

#Broker settings to confirm

Make sure the process supervisor gives Kafka time to shut down cleanly. If systemd or Kubernetes kills the JVM part-way through a controlled shutdown, the log has to be recovered at startup:

# /etc/systemd/system/kafka.service.d/override.conf
[Service]
TimeoutStopSec=300

On Kubernetes, set terminationGracePeriodSeconds to a similar value and use a PodDisruptionBudget with maxUnavailable: 1. Operators such as Strimzi run their own gated roll. Use theirs rather than deleting pods by hand.

#Order of nodes

1. Controllers (KRaft). Restart the followers first and the current quorum leader last, so the quorum goes through a single election. Get the leader from describe --status (LeaderId).
2. Brokers. Restart the broker that is the active controller last. In ZooKeeper mode that is the broker in /controller. In combined mode, a node is both broker and controller.
3. For a version upgrade, change the binaries on every node first. Finalize metadata.version only after the whole cluster has run the new version long enough that you won't roll back.

#Per-node procedure

#Step 1: confirm the cluster is clean

Run the pre-flight checks again. Never start a node while under-replicated partitions are non-zero from a previous step.

#Step 2: stop with a controlled shutdown

sudo systemctl stop kafka
journalctl -u kafka --since "-10 min" | grep -iE "controlled shutdown|shutdown completed|error"

A broker with thousands of leaders can take a minute or more to move them all. If the stop hits the timeout, find out why before you continue. The usual reasons are that the controller is unreachable or other replicas are out of sync.

#Step 3: apply the change and start

sudo systemctl start kafka
journalctl -u kafka -f | grep -iE "recover|started|error"

Log lines that mention recovering segments mean the previous stop was not clean. Recovery time scales with segment size and partition count.

#Step 4: gate on recovery

Wait until the restarted broker is back in every ISR:

until [ -z "$(kafka-topics.sh --bootstrap-server $BS --describe --under-replicated-partitions)" ]; do
  echo "$(date +%T) URP: $(kafka-topics.sh --bootstrap-server $BS --describe --under-replicated-partitions | wc -l)"
  sleep 15
done

For a controller, gate on describe --replication instead. The restarted voter should show Lag near zero and a recent LastCaughtUpTimestamp.

On each broker, track the kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions and kafka.controller:type=KafkaController,name=OfflinePartitionsCount metrics. Both must return to 0.

#Step 5: soak, then the next node

Wait a few minutes. Check producer error rates and consumer lag on the busiest topics, then move to the next node.

#After the last node

Leadership is now skewed toward the brokers you restarted first. If auto leader rebalancing is off, or you don't want to wait for its check interval (leader.imbalance.check.interval.seconds), restore preferred leaders yourself:

kafka-leader-election.sh --bootstrap-server $BS \
  --election-type PREFERRED --all-topic-partitions

Then confirm the change took effect:

kafka-configs.sh --bootstrap-server $BS --describe --entity-type brokers --entity-name 1 --all \
  | grep -E "<the setting you changed>"
kafka-broker-api-versions.sh --bootstrap-server $BS | head

#When it goes wrong

Caution

If a second broker fails during the roll, stop. Do not restart anything else until under-replicated and offline partitions are back to zero. With RF=3, two brokers down at the same time is when acks=all producers stop and unclean elections start to look tempting.

#Prevent it next time

• Script the gate: the next node should only start when URP is 0, and the script should abort on any offline partition.
• Ban RF=1 for production topics. A create-topic policy, or IaC review, catches them.
• Keep min.insync.replicas one below the replication factor, so a single restart never blocks writes.
• Track restart duration per broker. A sudden jump usually means unclean shutdowns or a growing partition count.