Kafka consumer rebalance storms: max.poll.interval.ms, cooperative rebalancing, static membership and the new consumer protocol

#Symptoms

• Logs full of Revoking previously assigned partitions, (Re-)joining group, Successfully joined group with generation N where N climbs quickly.
• Consumer lag grows in waves; throughput drops to zero during each rebalance.
• Messages processed twice after rebalances (offsets not committed before revocation).
• CommitFailedException: ... the group has already rebalanced and assigned the partitions to another member.

#Why consumers leave the group

A consumer is removed and a rebalance starts when:

Find which one from the client logs. The Java client logs, for example:

consumer poll timeout has expired. This means the time between subsequent calls to poll()
was longer than the configured max.poll.interval.ms ... sending LeaveGroup request

#Step 1: look at the group

kafka-consumer-groups.sh --bootstrap-server broker1:9092 --command-config client.properties \
  --describe --group orders-svc --state

kafka-consumer-groups.sh --bootstrap-server broker1:9092 --command-config client.properties \
  --describe --group orders-svc --members --verbose

State alternating between PreparingRebalance/CompletingRebalance and Stable, and member IDs that keep changing, confirm the storm.

#Step 2: slow processing (max.poll.interval.ms)

If a batch from poll() takes longer than max.poll.interval.ms to process, the consumer is kicked out.

max.poll.records=100
max.poll.interval.ms=600000

#Step 3: heartbeats (session.timeout.ms)

Heartbeats run on a background thread, so slow processing doesn't stop them, but long GC pauses, CPU throttling in containers, or network issues do.

• Check GC logs and container CPU throttling (container_cpu_cfs_throttled_seconds_total).
• Keep heartbeat.interval.ms at about a third of session.timeout.ms.
• Raise session.timeout.ms moderately if pauses are short and unavoidable (the broker limits the range with group.min/max.session.timeout.ms).

#Step 4: stop "stop-the-world" rebalances

With the classic eager protocol, every member gives up all partitions on each rebalance. Use cooperative rebalancing so only moved partitions are revoked:

partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Switching a live group from eager to cooperative needs a two-step rolling deployment (first list both assignors, then remove the eager one). See the client documentation for the safe upgrade path.

#Step 5: rolling restarts and Kubernetes

Every pod restart triggers two rebalances (leave and join). A rolling deploy of 20 pods = 40 rebalances.

Static membership gives each consumer a stable identity, so a restart within session.timeout.ms doesn't trigger a rebalance:

group.instance.id=${POD_NAME}        # must be unique and stable per instance (StatefulSet names work well)
session.timeout.ms=120000            # long enough to cover a restart

Trade-off: if a member really dies, its partitions aren't reassigned until the session timeout expires.

#Step 6: the new consumer group protocol (KIP-848)

Kafka 4.0 made the new consumer rebalance protocol generally available. The broker computes assignments and members reconcile incrementally, without a group-wide synchronisation barrier, which removes most stop-the-world behaviour.

group.protocol=consumer

It needs brokers and clients that support it (Kafka 4.0+ brokers in KRaft mode, recent Java client; other clients are catching up). Some classic settings, such as client-side partition.assignment.strategy, don't apply in this mode; assignment is configured on the broker side. Test in non-production first.

#Step 7: avoid duplicate processing during rebalances

• Commit offsets for processed records in onPartitionsRevoked (classic protocol) before partitions move.
• Make processing idempotent (upserts, dedup keys), because at-least-once delivery means some duplicates are always possible.

consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
  public void onPartitionsRevoked(Collection<TopicPartition> parts) { consumer.commitSync(currentOffsets(parts)); }
  public void onPartitionsAssigned(Collection<TopicPartition> parts) { }
});

#Decision table

#Prevent it next time

• Alert on consumer group rebalance rate (client metric rebalance-rate-per-hour in the Java client) and on lag.
• Use StatefulSets with static membership for long-running consumers on Kubernetes.
• Load test processing time per batch at peak, and set max.poll.records from that.
• Plan the move to the new consumer protocol once your clients support it.