Kafka compacted topic keeps growing: log cleaner stopped, active segments and tombstones that never go away

#Symptom

• A compacted topic (changelog, __consumer_offsets, CDC "latest state" topic) grows without limit.
• Consumers reading from the start see many old versions of the same key.
• Deleted keys keep coming back in rebuilds because tombstones never disappear, or disappear too early.
• Broker disk alerts on one or two brokers only.

#How compaction works, briefly

• Each partition is a series of segments. The newest one is the active segment. It is never compacted.
• The log cleaner (a pool of background threads on each broker) picks the partition with the highest dirty ratio (uncompacted bytes / total bytes), and rewrites closed segments keeping only the latest value per key.
• A tombstone (key with null value) deletes the key. It is kept for delete.retention.ms after the segment is cleaned, so slow consumers can see it, then removed.

#Step 1: is the log cleaner alive?

Check the broker's log-cleaner.log (in the Kafka log directory):

grep -E "ERROR|stopped|uncleanable" /var/log/kafka/log-cleaner.log | tail -20

And these JMX metrics:

If a cleaner thread hits an error on a partition (corrupt batch, out of memory for the dedupe buffer), older versions stopped the whole thread; newer versions mark the partition uncleanable and continue. Either way that partition stops being compacted.

#Step 2: fix a dead cleaner or uncleanable partition

Common causes:

• Dedupe buffer too small for a partition with a huge number of distinct keys in one segment. Raise log.cleaner.dedupe.buffer.size (shared across log.cleaner.threads).
• Corrupt segment after an unclean shutdown. The log shows the file name. With replication factor 3, the usual fix is to stop the broker, move the affected partition directory aside, and let it re-replicate from the leader. Do this one broker at a time and only when other replicas are in sync.
• Messages without keys in a compacted topic. Producers must send a key; keyless records make compaction fail for that partition.

A broker restart restarts the cleaner threads. That's a valid short-term action once the cause is fixed.

Caution

Don't delete segment files on a running broker, and never on the only in-sync replica. Check kafka-topics.sh --describe --under-replicated-partitions first.

#Step 3: segments that never roll

If the cleaner is healthy but the topic still grows, the data may be sitting in the active segment. For a low-volume topic with segment.bytes = 1 GB, a segment can stay active for months.

kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics \
  --entity-name customer-state --describe

Set segment.ms so segments close regularly:

kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics \
  --entity-name customer-state --alter --add-config segment.ms=3600000

#Step 4: tune when compaction runs

For a GDPR-style "delete must be gone within N days" requirement, set max.compaction.lag.ms plus delete.retention.ms below N, and make sure segment.ms doesn't keep data in the active segment longer than that.

#Step 5: tombstones

• Tombstones never disappear: the segment holding them hasn't been cleaned yet (active segment, dirty ratio not reached), or delete.retention.ms is very large.
• Deleted keys come back: a consumer rebuilding state was slower than delete.retention.ms, so it never saw the tombstone but did see an older value in another not-yet-compacted segment. Increase delete.retention.ms above your longest rebuild time.

Check disk use per partition to see whether the problem is one partition or all:

kafka-log-dirs.sh --bootstrap-server broker1:9092 --describe --topic-list customer-state \
  | grep -o '"partition":"[^"]*","size":[0-9]*' | sort -t: -k3 -n | tail

#Prevent it next time

• Alert on uncleanable-partitions-count > 0 and on time-since-last-run-ms above an hour.
• Set segment.ms on every compacted topic at creation time.
• Enforce keys in producers for compacted topics (schema and client checks).
• Related: Kafka broker disk full, Kafka retention and tiered storage.