Symptoms
Producers with acks=all log errors like:
org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.
org.apache.kafka.common.errors.NotEnoughReplicasAfterAppendException: Messages are written to the log, but to fewer in-sync replicas than required.
librdkafka clients show Broker: Not enough in-sync replicas. Throughput drops to zero for some partitions; others keep working.
Why it happens
For each partition:
- Replication factor (RF): how many copies exist, for example 3.
- ISR (in-sync replicas): the replicas currently caught up with the leader.
- min.insync.replicas: with
acks=all, the leader only accepts a write if the ISR has at least this many members, commonly 2.
If the ISR shrinks below min.insync.replicas, the leader refuses writes rather than risk acknowledging data that exists on too few brokers. That's working as designed: it protects you from data loss.
| RF | min.insync.replicas | Brokers that can be down/lagging before writes stop |
|---|---|---|
| 3 | 2 | 1 |
| 3 | 3 | 0 (any blip stops writes) |
| 2 | 2 | 0 |
| 3 | 1 | 2 (but acks=all no longer guarantees 2 copies) |
Step 1: Find the affected partitions
# Partitions whose ISR is below min.insync.replicas
kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-min-isr-partitions
# Partitions with fewer in-sync replicas than RF
kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions
# Partitions with no leader
kafka-topics.sh --bootstrap-server broker1:9092 --describe --unavailable-partitions
Look at the Replicas and Isr columns. The broker IDs that appear in Replicas but not in Isr are the problem brokers.
Check the topic's settings:
kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics --entity-name orders --describe --all \
| grep -E 'min.insync.replicas|unclean.leader'
Step 2: Why are replicas out of ISR?
A follower drops out of the ISR if it doesn't catch up within replica.lag.time.max.ms (default 30 s in recent versions). Common causes:
| Cause | How to see it |
|---|---|
| Broker down or restarting | Broker missing from cluster metadata; process/pod not running |
| Disk full or failed log directory | Broker log: Error while writing to checkpoint file, offline log dirs; df -h |
| Slow disk / I/O saturation | High await in iostat; replica fetcher lag grows |
| Network problems between brokers | Fetcher errors, timeouts in broker logs |
| Long GC pauses | GC logs with multi-second pauses |
| Too few replica fetcher threads for partition count | Lag grows evenly on a busy broker |
Useful broker metrics (JMX):
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitionskafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCountkafka.server:type=ReplicaManager,name=IsrShrinksPerSec/IsrExpandsPerSeckafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica
On MSK, the same appear in CloudWatch as UnderReplicatedPartitions, UnderMinIsrPartitionCount and so on.
Step 3: Fix the broker, let the ISR expand
- Bring the failed broker back (restart the service/pod, replace the node, free disk).
- Watch the ISR grow back as the follower catches up:
watch -n 10 "kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-min-isr-partitions | wc -l"
- Once all partitions are back to full ISR, run a preferred leader election so leadership is balanced again:
kafka-leader-election.sh --bootstrap-server broker1:9092 --election-type PREFERRED --all-topic-partitions
If a broker is gone for good, reassign its partitions to healthy brokers with kafka-reassign-partitions.sh (generate a plan, review it, execute with a --throttle, then --verify).
Tip
When a broker rejoins after a long outage, catching up can saturate disks and network. Use replication throttles (
leader.replication.throttled.rate/follower.replication.throttled.rate) during big catch-ups so client traffic still gets through.
Step 4: The shortcuts, and why to avoid them
Lowering min.insync.replicas to 1 gets writes flowing immediately, but with RF=3 and one replica in the ISR you are acknowledging writes stored on a single disk. If that broker dies, those messages are gone.
# Only as a deliberate, time-boxed decision, with the business owner's agreement
kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics --entity-name orders \
--alter --add-config min.insync.replicas=1
# Revert as soon as ISR recovers
kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics --entity-name orders \
--alter --delete-config min.insync.replicas
Enabling unclean.leader.election.enable=true lets an out-of-sync replica become leader when no in-sync replica is available. It restores availability and throws away messages the old leader had that the new one doesn't.
Caution
Both changes trade durability for availability. For payment, order or audit topics, waiting for the broker to recover is usually the right call. Write down who approved the change and when it was reverted.
Step 5: Producer behaviour while you fix it
- Producers retry retriable errors (
NotEnoughReplicasis retriable) untildelivery.timeout.ms(default 2 minutes) expires. Raise it for critical producers so short outages don't drop messages. - Keep
enable.idempotence=true(default in recent clients) so retries don't create duplicates. - Make sure the application handles send failures (callback/future errors) instead of logging and continuing.
Prevent it next time
- Use RF=3 and
min.insync.replicas=2for important topics, and spread replicas across racks/AZs (broker.rack). - Alert on
UnderMinIsrPartitionCount > 0(page) andUnderReplicatedPartitions > 0for more than a few minutes (warn). - Keep at least 30-40% disk headroom on brokers and alert on disk usage; full disks are the most common cause.
- Do rolling restarts one broker at a time and wait for zero under-replicated partitions before the next.