Home / Data Platform / Streaming & CDC

Data PlatformRunbookAdvanced

Kafka broker disk full: emergency recovery and retention tuning

Find which topics fill the broker disks, free space safely by lowering retention, recover an offline broker, and prevent it with quotas, alerts and tiered storage.

Deepak Bhardwaj2 min read

Applies toApache Kafka 3.x (ZooKeeper or KRaft), Confluent Platform, Amazon MSK

Symptoms

  • Broker log: java.io.IOException: No space left on device, then the broker shuts down.
  • Partitions show as offline or under-replicated; producers get NOT_LEADER_OR_FOLLOWER or time out.

Warning

Never delete files in the Kafka log directory by hand while the broker runs, and don't delete the newest segments of any partition. That causes data loss and can corrupt indexes. Free space through retention instead.

1. Which topics use the space?

kafka-log-dirs.sh --bootstrap-server broker1:9092 --describe --broker-list 1 \
  | grep '^{' | jq -r '.brokers[].logDirs[].partitions[] | "\(.size) \(.partition)"' \
  | sort -nr | head -20

du -sh /var/lib/kafka/data/* | sort -h | tail -20      # on the broker host

2. Free space: lower retention on the biggest topic temporarily

kafka-configs.sh --bootstrap-server broker1:9092 --alter --entity-type topics \
  --entity-name clickstream --add-config retention.ms=21600000      # 6 hours

The log cleaner deletes closed segments after the next check (log.retention.check.interval.ms, default 5 minutes). Make sure consumers of that topic aren't lagging further than the new retention, or they'll lose messages.

Put retention back afterwards:

kafka-configs.sh --bootstrap-server broker1:9092 --alter --entity-type topics \
  --entity-name clickstream --delete-config retention.ms

3. If the broker is already down

  1. Add disk (extend the volume and grow the filesystem), or add a new log dir to log.dirs.
  2. Start the broker. It recovers its segments, which can take a while after an unclean shutdown.
  3. Then apply the retention change so it doesn't fill again.

4. Check cluster health

kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions
kafka-topics.sh --bootstrap-server broker1:9092 --describe --unavailable-partitions

5. Rebalance uneven disks

One broker full while others are half empty → partition placement is uneven. Use kafka-reassign-partitions.sh (or Cruise Control) with a throttle so replication traffic doesn't hurt production:

kafka-reassign-partitions.sh --bootstrap-server broker1:9092 \
  --reassignment-json-file plan.json --execute --throttle 50000000

Prevent it

  • Alert on disk usage at 70% and 85% per broker.
  • Set retention.bytes (per partition) on high-volume topics, not only retention.ms.
  • Use producer quotas for noisy clients.
  • Use tiered storage (KIP-405, MSK tiered storage, Confluent) to move old segments to object storage.

Tip

Compute capacity as: daily ingest × retention days × replication factor ÷ number of brokers, plus 30% headroom. Most "surprise" disk-full incidents fail that sum.

Related fixes