Add partitions to a live Kafka topic: a runbook for key ordering, consumer offsets, compacted topics and Kafka Streams

A consumer group cannot keep up because the topic has 12 partitions and you cannot run more than 12 consumers. Adding partitions is one command and takes seconds. It is also irreversible, and on keyed topics it changes where every key goes from that moment on. This runbook covers the decision, the preparation that prevents data loss, the change and the checks afterwards.

Caution

Kafka cannot reduce the partition count of a topic. The only way back is a new topic and a migration. Agree the target count before you run anything.

#Step 1: decide whether adding partitions is safe

The default partitioner maps a key with hash(key) % partitionCount. Change the count and most keys map to a different partition. Records already written stay where they are.

If any "No" applies, skip to "Alternative: migrate to a new topic" below.

#Step 2: record the current state

BS=broker1:9092
TOPIC=orders

kafka-topics.sh --bootstrap-server $BS --describe --topic $TOPIC

kafka-configs.sh --bootstrap-server $BS --describe \
  --entity-type topics --entity-name $TOPIC --all | grep -E "cleanup.policy|retention|min.insync"

# Every group reading the topic, with lag per partition
for G in $(kafka-consumer-groups.sh --bootstrap-server $BS --list); do
  kafka-consumer-groups.sh --bootstrap-server $BS --describe --group "$G" 2>/dev/null \
    | awk -v t=$TOPIC '$2==t' | sed "s/^/$G /"
done

Save the output in the change ticket. You need the group list for Step 3 and the lag for the ordering decision.

Check broker capacity too: each new partition adds replicas, open files and leader work. Compare against your per-broker partition limits (check current guidance for your version or managed service; Amazon MSK publishes per-instance-size recommendations).

#Step 3: protect consumers from skipping records

This is the step most teams miss.

Consumers discover new partitions on their next metadata refresh, controlled by metadata.max.age.ms (default 5 minutes). Producers discover them the same way and may write to a new partition before any consumer has an offset there. When the group is finally assigned the new partition, it has no committed offset, so it uses auto.offset.reset. With the default latest, it starts at the end and the records written in that gap are never read.

For every group in Step 2:

If you cannot redeploy a consumer, pre-create offsets at 0 for the new partitions right after the alter and before producers find them, with the group stopped:

# Group must be inactive (no running members) for reset-offsets to apply
kafka-consumer-groups.sh --bootstrap-server $BS --group billing-consumer \
  --reset-offsets --topic $TOPIC:12,13,14,15,16,17,18,19,20,21,22,23 \
  --to-earliest --execute

The topic:partition list syntax is supported by kafka-consumer-groups.sh; run with --dry-run first to confirm the plan.

#Step 4: drain lag on ordered topics

For keyed topics where order matters:

1. Scale consumers so lag reaches zero, or near zero, on every partition.
2. If ordering must be strict, pause the producers (or stop the upstream connector) at a clean point.
3. Confirm lag is zero for every group with the loop from Step 2.

Once lag is zero, the old records for every key have been consumed, so there is no older backlog in the old partition for a consumer of the new partition to overtake.

#Step 5: add the partitions

kafka-topics.sh --bootstrap-server $BS --alter --topic $TOPIC --partitions 24
kafka-topics.sh --bootstrap-server $BS --describe --topic $TOPIC

Check the describe output: 24 partitions, each with the expected replication factor and a full ISR. New partition replicas are placed by the controller; if you use rack awareness or have uneven brokers, review placement and rebalance with kafka-reassign-partitions.sh afterwards.

Resume producers. They switch to the new count within metadata.max.age.ms; restart them if you need it immediate.

#Step 6: verify

# All partitions receiving data
kafka-get-offsets.sh --bootstrap-server $BS --topic $TOPIC

kafka-get-offsets.sh exists in Kafka 3.0+; on older tools use kafka-run-class.sh kafka.tools.GetOffsetShell.

Then confirm:

• Every consumer group shows members assigned to the new partitions and lag on them is moving.
• Consumer count can now be raised to the new partition count.
• Downstream row counts for the change window match the source (no gap from Step 3).
• Key distribution looks even: no new partition stuck at zero offsets.

#Mixed-language producers

Java clients hash keys with murmur2. librdkafka-based clients (Python, Go, .NET, C) default to partitioner=consistent_random, which uses CRC32. If both write the same keys, a key already maps to two partitions. Set partitioner=murmur2_random in librdkafka clients so mappings match. Fix this before adding partitions, not after.

#Alternative: migrate to a new topic

For compacted topics, Kafka Streams topics, or keyed topics where you cannot pause producers, create a new topic and move to it.

1. Create orders.v2 with the target partition count and the same configs.
2. Copy existing data with a small consumer-producer job that keeps keys, or for compacted topics, re-publish the latest value per key so the new topic is correctly keyed.
3. Switch producers to orders.v2 (dual-write briefly if consumers need overlap).
4. Move each consumer group once it has finished the old topic, starting from earliest on the new one, with idempotent processing to absorb overlap.
5. Keep the old topic read-only until retention expires or you have validated, then delete it.

For Kafka Streams, deploy the topology against the new input with a new application.id, or stop the app and run kafka-streams-application-reset.sh for the old ID before restarting. Internal changelog and repartition topics are then rebuilt with the right partition counts.

Warning

Never alter the partition count of a Kafka Streams internal topic (<application.id>-...-changelog or -repartition) by hand. State stores map one-to-one to partitions and the app will fail or restore the wrong state.

#Rollback

There is no in-place rollback for partition count. If the change caused problems:

• Consumer gaps: reset the affected groups on the new partitions to earliest and reprocess with idempotent sinks.
• Ordering problems: pause producers, drain, and accept the new mapping; or migrate to a new topic with the original count.

#Prevent it next time

• Size partitions at creation for two to three years of growth in consumer parallelism, within broker limits, so you rarely need this runbook.
• Default consumers to auto.offset.reset=earliest unless there is a clear reason not to, and make processing idempotent.
• Standardise the partitioner across client languages.
• Mark compacted and Streams topics in your topic catalog as "migrate, do not alter" so on-call does not run --alter on them under pressure.