Home / Data Platform / Streaming & CDC

Data PlatformTroubleshootingAdvanced

Kafka consumer lag keeps growing: a troubleshooting guide

Measure lag per partition, tell slow processing from rebalance storms and hot partitions, and reset offsets safely when you need to.

Deepak Bhardwaj3 min read

Applies toApache Kafka 3.x, Amazon MSK, Azure Event Hubs (Kafka endpoint), OCI Streaming, Confluent

Measure the lag

kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group orders-processor

GROUP            TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG     CONSUMER-ID
orders-processor orders  0          1203400         1203460         60      consumer-1-...
orders-processor orders  1          998000          1450000         452000  consumer-2-...
orders-processor orders  2          1201100         1201190         90      -

How to read it:

  • Lag on every partition → the consumers can't keep up overall.
  • Lag on one partition → a hot key, or one slow or stuck consumer.
  • CONSUMER-ID is - → no consumer owns that partition. The group is rebalancing, or you have fewer consumers than you think.

Cause 1: processing is too slow

  • Time how long each batch takes. Slow downstream calls (a DB write or HTTP call per record) are the usual cause.
  • Batch the downstream writes, and make processing asynchronous within a poll loop where ordering allows it.
  • Add consumers, up to the number of partitions. Consumers beyond that sit idle.

Cause 2: rebalance storms

If processing one poll's records takes longer than max.poll.interval.ms (default 5 minutes), the consumer is removed from the group. That triggers a rebalance, and the records are processed again.

Look for this in consumer logs: consumer poll timeout has expired or Member ... leaving group.

Fixes:

max.poll.records=200                 # smaller batches per poll
max.poll.interval.ms=600000          # more time per batch, if processing is legitimately slow
group.protocol=consumer              # new rebalance protocol (KIP-848) on Kafka 4.x
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Cooperative rebalancing stops the entire group from pausing every time one consumer joins or leaves.

Cause 3: hot partitions

kafka-get-offsets.sh --bootstrap-server broker:9092 --topic orders   # end offset per partition; sample twice to get a rate

If one partition receives far more messages than the others, the producer key is skewed (for example, a single big customer ID). Options: add a sub-key or salt to the key, or process that key on a dedicated path.

Cause 4: the broker side

  • Under-replicated partitions or a broker restart can make fetches slow.
  • Throttled quotas (fetch-throttle-time-avg above 0) on managed services. On Event Hubs, check throughput units / processing units and throttled-request metrics. On MSK, check BytesOutPerSec against broker limits.

Resetting offsets (carefully)

Stop all consumers in the group first, because a reset fails on an active group. Then do a dry run:

kafka-consumer-groups.sh --bootstrap-server broker:9092 --group orders-processor --topic orders \
  --reset-offsets --to-datetime 2026-10-22T06:00:00.000 --dry-run
# review the output, then replace --dry-run with --execute

Caution

--to-latest skips all unprocessed messages, and they're lost to that consumer group. Only use it when the business has agreed that the data can be dropped or will be reprocessed another way.

Alert on the right signal

Use lag in time (how old the oldest unprocessed message is), not only message count. A lag of 50,000 messages is fine at 100k msg/s and very bad at 10 msg/s. Burrow, Kafka Lag Exporter, or your cloud provider's consumer-lag metric can track this.

Related fixes