1. Measure size per topic
kafka-log-dirs.sh --bootstrap-server broker:9092 --describe | grep '^{' \
| jq -r '[.brokers[].logDirs[].partitions[] | {t: (.partition | sub("-[0-9]+$"; "")), s: .size}]
| group_by(.t) | map({t: .[0].t, gb: ((map(.s) | add) / 1e9)}) | sort_by(-.gb)[:20][]
| "\(.gb | floor) GB\t\(.t)"'
Sizes include all replicas across brokers.
(Or use your monitoring: per-topic size metrics in MSK/Confluent/Prometheus exporters.)
Compare each big topic's size with its retention and its consumers' real needs.
2. Retention by use case
| Topic type | Retention / cleanup |
|---|---|
| Event stream consumed in near real time | 1–7 days (retention.ms) |
| Topic consumers may replay after incidents | Longer, ideally on tiered storage |
| Latest state per key (customer profile, config) | cleanup.policy=compact |
| Changelog with history needed for a while | compact,delete with retention |
| Audit/compliance | Archive to object storage (sink connector), shorter Kafka retention |
kafka-configs.sh --bootstrap-server broker:9092 --alter --entity-type topics --entity-name clickstream \
--add-config retention.ms=259200000 # 3 days
kafka-configs.sh --bootstrap-server broker:9092 --alter --entity-type topics --entity-name customer-profile \
--add-config cleanup.policy=compact,min.cleanable.dirty.ratio=0.3
Retention applies to closed segments. segment.ms/segment.bytes affect how promptly old data is deleted.
3. Tiered storage for long retention
Tiered storage keeps recent data on broker disks and older segments in object storage, so long retention doesn't need huge broker disks:
# broker (plus a RemoteStorageManager plugin for your object store)
remote.log.storage.system.enable=true
# topic
remote.storage.enable=true
local.retention.ms=86400000 # 1 day on local disk
retention.ms=2592000000 # 30 days total
Managed services (MSK, Confluent) expose this as a topic or cluster setting. Check feature limitations (e.g. compacted topics aren't supported for tiering) and how remote reads affect catch-up consumers.
4. Compression
Producer-side compression (zstd/lz4) shrinks data on disk and network. Keep topic compression.type=producer to store what producers send, unless you need to force a codec.
5. Partition counts
| Too few | Too many |
|---|---|
| Consumer parallelism limited | More open files, memory, metadata, slower recovery and rebalances |
Size partitions for target throughput and consumer parallelism, with some headroom. Partitions can be increased later (this changes key-to-partition mapping), never decreased. Delete abandoned topics:
kafka-consumer-groups.sh --bootstrap-server broker:9092 --list # groups still alive?
kafka-topics.sh --bootstrap-server broker:9092 --describe --topic old-feed
Check for producers (bytes-in) and consumers before deleting a topic.
6. Replication factor by value
RF 3 with min.insync.replicas=2 for important topics. RF 2 or less only for data you can regenerate (and document it). Don't reduce RF on business data to save disk.
7. Track it
Monthly: total disk per broker, top 10 topics by size, topics with no consumers, retention settings vs agreed policy.
Caution
Lowering retention deletes data on brokers soon after the change. Make sure no consumer is behind, and that downstream archives have the data, before shortening retention on a busy topic.
Checklist
- Size per topic measured
- Retention/compaction per use case
- Tiered storage for long retention
- Producer compression on
- Partition counts reviewed; abandoned topics removed
- RF kept for business data; monthly review