Sev1: Kafka consumer lag exploding — rebalance storms, max.poll.interval.ms and slow processing

#Symptoms

• Lag climbs steadily on one consumer group while producers are normal.
• Consumer logs repeat Attempt to heartbeat failed since group is rebalancing, Member ... sending LeaveGroup request ... consumer poll timeout has expired, or CommitFailedException.
• Throughput drops to near zero during each rebalance.

#1. Measure lag and membership

kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group orders-enricher
kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group orders-enricher --members --verbose

#2. Rebalance storm: poll timeout exceeded

A consumer that takes longer than max.poll.interval.ms (default 300000 ms) between poll() calls is kicked out of the group. Its partitions move, another consumer gets the same slow batch, and the cycle repeats.

Typical trigger: a downstream dependency (database, API) slowed down, so each batch takes longer.

Fixes, in order of speed:

# Process fewer records per poll so each loop finishes in time
max.poll.records=100
# Give slow batches more time (only if processing is legitimately slow)
max.poll.interval.ms=600000

• Fix or isolate the slow dependency (timeouts, circuit breaker).
• Don't do blocking retries with long sleeps inside the poll loop.

#3. Reduce rebalance cost

# Cooperative rebalancing: only moved partitions stop, not all
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
# Static membership: restarts within session timeout don't trigger rebalance
group.instance.id=orders-enricher-${HOSTNAME}
session.timeout.ms=45000

Warning

Changing assignment strategy needs a rolling upgrade procedure: add the new assignor alongside the old one first, roll all consumers, then remove the old one. Switching in one step can fail group joins.

On Kafka 4.x with the new consumer group protocol (KIP-848), rebalancing is server-side and much cheaper; check your client and broker versions before relying on it.

#4. Not enough capacity

• Consumers in a group can't exceed the number of partitions. If you already have one consumer per partition, adding more does nothing.
• Increase partitions only if ordering by key allows it; it changes key-to-partition mapping for new messages.

kafka-topics.sh --bootstrap-server broker1:9092 --describe --topic orders
kafka-topics.sh --bootstrap-server broker1:9092 --alter --topic orders --partitions 48

• Process in parallel inside each consumer (with care for ordering per key), or batch writes downstream.

#5. Hot partition

If one partition has most of the lag, a single key (customer, tenant) is producing much more than others. Short-term: give that consumer more resources. Long-term: change the partitioning key or add a salt for that key.

#6. Catching up safely

When the fix is in, lag drains at full speed and can overload downstream systems. Throttle with max.poll.records and downstream connection pools. If the business accepts skipping old data, resetting offsets is an option:

kafka-consumer-groups.sh --bootstrap-server broker1:9092 --group orders-enricher \
  --topic orders --reset-offsets --to-datetime 2026-10-31T08:00:00.000 --dry-run
# then --execute, with all consumers in the group stopped

Caution

Resetting offsets forward skips data permanently for that group. Get explicit sign-off and record the range skipped.

#Prevent it next time

• Alert on lag growth rate and on rebalance count per group, not just absolute lag.
• Use cooperative-sticky assignment and static membership.
• Set timeouts on every downstream call inside the poll loop.
• Load test consumers with production record sizes.