Sev1: Kafka broker disks almost full — emergency retention changes and partition moves without data loss

#Symptoms

• Disk usage alerts on one or more brokers' log directories.
• At 100% the broker marks the log directory offline or shuts down, leaving partitions under-replicated or offline.

#1. Find who is using the space

kafka-log-dirs.sh --bootstrap-server broker1:9092 --describe --broker-list 1,2,3 \
  | grep '^{' | jq -r '.brokers[] | .broker as $b | .logDirs[] | .partitions[] | "\($b) \(.partition) \(.size)"' \
  | sort -k3 -n -r | head -20

On the broker host:

df -h /var/lib/kafka
du -sh /var/lib/kafka/data/* | sort -h | tail -20

Typical causes:

#2. Buy time: lower retention on the biggest topics

kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics --entity-name clickstream \
  --describe
kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics --entity-name clickstream \
  --alter --add-config retention.ms=21600000   # 6 hours

Deletion happens per closed segment when the log cleaner runs, so it can take minutes. Make sure consumers of that topic are not lagging behind the new retention, or they will skip data.

kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --all-groups | grep clickstream

Caution

Lowering retention deletes data permanently once segments age out. Confirm with data owners and check consumer lag first. Never delete segment files by hand from the data directory while the broker runs.

You can also cap by size:

kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics --entity-name clickstream \
  --alter --add-config retention.bytes=50000000000   # per partition

#3. Stop the runaway producer

If one producer is flooding a topic, throttle it with a client quota:

kafka-configs.sh --bootstrap-server broker1:9092 --alter \
  --add-config 'producer_byte_rate=1048576' --entity-type clients --entity-name clickstream-producer

#4. Move partitions off the full broker

Generate a plan, review, then execute with a throttle so replication doesn't saturate the network:

cat > topics.json <<'EOF'
{"topics":[{"topic":"clickstream"}],"version":1}
EOF
kafka-reassign-partitions.sh --bootstrap-server broker1:9092 --topics-to-move-json-file topics.json \
  --broker-list 2,3,4 --generate > plan.txt
# save the proposed assignment as reassign.json, review it
kafka-reassign-partitions.sh --bootstrap-server broker1:9092 --reassignment-json-file reassign.json \
  --execute --throttle 50000000
kafka-reassign-partitions.sh --bootstrap-server broker1:9092 --reassignment-json-file reassign.json --verify

The --verify step also removes the throttle when done. Moving data needs free space on the target brokers.

#5. Add capacity

• Extend the volume (cloud disks can usually grow online), then grow the filesystem (xfs_growfs, resize2fs).
• Add another log directory to log.dirs (requires a broker restart) and move partitions there with kafka-reassign-partitions.sh using log_dirs in the plan.
• On MSK, increase broker storage (it can only grow, not shrink).

#6. If a broker already died

Clear space (extend disk), start the broker, and watch under-replicated partitions return to zero:

kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions

#Prevent it next time

• Alerts at 70% and 85% disk per broker; growth-rate alerts per topic.
• Require explicit retention on every topic; forbid retention.ms=-1 without approval.
• Producer quotas for high-volume clients.
• Consider tiered storage (Kafka 3.6+ / platform features) to move old segments to object storage.