Kafka messages duplicated or out of order after producer retries: idempotence, max.in.flight and OutOfOrderSequenceException

Consumers report the same event twice, or a "status=closed" event arriving before "status=open" for the same key. Nothing failed in the producer logs except a few retried batches during a broker restart. This is the classic retry problem: the producer resent a batch the broker had already written, or a retried batch landed after a later one.

The idempotent producer fixes both within a single partition and producer session. The catch is that it is often not actually on, or something else breaks the guarantee.

#How retries cause duplicates and reordering

• Duplicates: the broker writes a batch, the acknowledgement is lost (timeout, leader change, connection reset), and the producer resends. Without idempotence the broker writes it again.
• Reordering: with max.in.flight.requests.per.connection above 1 and no idempotence, batch 1 fails and is retried while batch 2 succeeds. Batch 1 is written after batch 2.

With idempotence, each producer gets a producer ID (PID) and stamps every batch with a per-partition sequence number. The broker keeps the recent sequence numbers per PID and partition, drops duplicates and rejects gaps, so up to 5 in-flight requests stay ordered.

#Check 1: is idempotence really on?

Since Kafka 3.0 the Java client defaults to enable.idempotence=true and acks=all. Two things commonly undo that:

Read the ProducerConfig values: block the Java client logs at startup and confirm the effective values, not what is in your config file. For librdkafka clients, set debug=eos briefly (debug contexts vary by version; check the CONFIGURATION.md of your build) to see PID acquisition.

Set it explicitly so a later change to acks fails loudly instead of silently disabling it:

enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5
retries=2147483647
delivery.timeout.ms=120000

For confluent-kafka-python:

from confluent_kafka import Producer

p = Producer({
    "bootstrap.servers": "broker1:9092",
    "enable.idempotence": True,   # forces acks=all and safe in-flight/retry values
    "linger.ms": 20,
})

#Check 2: is the broker tracking the producer?

kafka-transactions.sh can list producer state for a partition, including idempotent (non-transactional) producers:

kafka-transactions.sh --bootstrap-server broker1:9092 \
  describe-producers --topic orders --partition 0

You should see your producer IDs with recent last sequence and timestamp. No entries while the producer is writing means it is not idempotent.

The principal also needs permission. Older brokers required the IdempotentWrite cluster ACL; from Kafka 2.8 Write on a topic is enough. Mixed estates (old brokers, managed services with their own auth) can still reject PID allocation, which shows up as ClusterAuthorizationException at producer start.

#Check 3: where else duplicates come from

Idempotence only covers retries inside one producer instance for one partition. Duplicates still appear from:

Tip

Before blaming the producer, compare offsets. Two copies of a record at different offsets in the same partition means the producer wrote it twice. The same offset processed twice means the consumer reprocessed.

kafka-console-consumer.sh --bootstrap-server broker1:9092 --topic orders \
  --partition 0 --offset 1048000 --max-messages 200 \
  --property print.key=true --property print.offset=true --property print.timestamp=true

print.offset and print.partition properties exist in recent console consumer versions; older builds may lack them.

#OutOfOrderSequenceException

The broker received a batch whose sequence number is not the next expected one for that PID and partition. Typical causes:

• Data loss on the broker side: an unclean leader election or a leader with acks weaker than all lost the earlier batches, so the new leader has a gap.
• Records removed by retention or deleteRecords while the producer was still sending.
• Client bugs in old versions around retries during leader changes.

In current Java clients a non-transactional idempotent producer usually recovers by bumping its epoch and resetting sequences. If you see the exception surfaced to your callback:

1. Check the broker logs and controller events for leader elections at that time, and unclean.leader.election.enable on the topic.
2. Confirm min.insync.replicas is at least 2 with replication factor 3, and the producer uses acks=all.
3. Treat the batches in flight as possibly lost; re-send them from the source with keys that consumers can dedupe.

kafka-configs.sh --bootstrap-server broker1:9092 --describe \
  --entity-type topics --entity-name orders --all | \
  grep -E "unclean.leader.election.enable|min.insync.replicas"

Caution

unclean.leader.election.enable=true trades durability for availability. If it is on for a topic that needs ordering and no-loss guarantees, turn it off, and expect sequence errors after any unclean election.

#UnknownProducerIdException and producer ID expiry

Brokers drop producer state that has been idle longer than producer.id.expiration.ms (default one day in recent versions; check your broker's docs), and state for producers whose records have all been removed by retention. A producer that writes rarely, or to a topic with very short retention, may find the broker no longer knows its PID. Recent clients handle this by obtaining a new epoch; very old clients surface UnknownProducerIdException. Upgrade the client before tuning broker settings.

#Ordering across producers and partitions

Idempotence orders records from one producer to one partition. It does not order:

• Records for the same key from two producer instances. Route each key through one producer, or carry a version or timestamp and resolve order in the consumer.
• Records across partitions. Ordering is per partition only.
• Records after a partition count change. The key-to-partition mapping changes, so old and new records for a key live in different partitions.

Java clients hash keys with murmur2. librdkafka's default partitioner is consistent_random (CRC32). If Java and non-Java producers write the same keys, set partitioner=murmur2_random on the librdkafka side so keys map to the same partitions.

#Prevent it next time

• Set enable.idempotence=true and acks=all explicitly in every producer config template, and fail the build if acks is overridden.
• Pin client versions past the 3.0/3.1 bug and above 3.2 for Java.
• Disable unclean leader election on ordered topics; use replication factor 3 and min.insync.replicas=2.
• Standardise the partitioner across languages.
• Make consumers idempotent anyway: upsert by key with a version column, because consumer-side reprocessing is the most common source of duplicates.