Kafka transactions gone wrong: ProducerFencedException, hanging transactions and read_committed consumers stuck behind the LSO

#Symptoms

• Producers fail with ProducerFencedException or InvalidProducerEpochException and the application restarts in a loop.
• Consumers with isolation.level=read_committed show lag that never decreases on some partitions, while read_uncommitted consumers read fine.
• Partition size keeps growing, but the committed data stops at a fixed offset.

#How transactions block readers

A transactional producer writes records, then commits or aborts. Until the outcome is known, read_committed consumers can only read up to the Last Stable Offset (LSO): the first offset of the oldest open transaction on that partition.

If a transaction is never completed (a "hanging" transaction), the LSO stops moving. read_committed consumers on that partition are stuck behind it, forever.

Normally the transaction coordinator aborts transactions that exceed transaction.timeout.ms. Hanging transactions are rare edge cases (bugs in older client/broker versions, or a write that landed on a partition the coordinator didn't know about), but when they happen, they stop a pipeline.

#Step 1: is it fencing or hanging?

#Step 2: fencing (ProducerFencedException)

Fencing is a safety feature: two producers must never write in the same transactional "session". It happens when:

• Two instances are configured with the same transactional.id (copy-pasted config, two replicas of a deployment with a fixed ID).
• A restarted instance comes back while the old one is still alive (zombie), for example during a slow Kubernetes rollout.
• A transaction ran longer than transaction.timeout.ms, was aborted by the coordinator, and the producer later tried to continue.

Fixes:

• Give every producer instance a unique, stable transactional.id (for example orders-processor-${POD_ORDINAL} in a StatefulSet).
• Treat ProducerFencedException as fatal for that producer: close it and create a new one; don't retry the same instance.
• Keep transactions short. Set transaction.timeout.ms (default 60000) above your longest transaction but below the broker's transaction.max.timeout.ms (default 900000).

#Step 3: find hanging transactions

Kafka 3.0+ ships kafka-transactions.sh (KIP-664):

# List transactional producers and their state
kafka-transactions.sh --bootstrap-server broker1:9092 --command-config admin.properties list

# Describe one transactional.id
kafka-transactions.sh --bootstrap-server broker1:9092 --command-config admin.properties \
  describe --transactional-id orders-processor-0

# Find transactions open for longer than they should be, per broker
kafka-transactions.sh --bootstrap-server broker1:9092 --command-config admin.properties \
  find-hanging --broker-id 1 --max-transaction-timeout 15

describe-producers shows the open transaction start offset per producer on a partition:

kafka-transactions.sh --bootstrap-server broker1:9092 --command-config admin.properties \
  describe-producers --topic orders --partition 3

An entry with a CurrentTransactionStartOffset far behind the log end, and old timestamps, is your hanging transaction.

#Step 4: abort it

kafka-transactions.sh --bootstrap-server broker1:9092 --command-config admin.properties \
  abort --topic orders --partition 3 --start-offset 1048576

The abort writes a marker that closes the transaction; the LSO moves forward and read_committed consumers continue. Records in the aborted transaction are skipped by read_committed consumers.

Caution

Aborting discards the transaction's records for read_committed readers. Make sure the producing application has really given up on it (or is gone), and that the data can be re-produced if needed. The admin user needs the right ACLs (Write on the topic, cluster permissions as documented for the tool).

#Step 5: Kafka Streams and Connect

• Kafka Streams with processing.guarantee=exactly_once_v2 manages transactional IDs itself. Fencing errors usually mean two instances share an application.id and state directories in a bad way, or a thread stalled past the transaction timeout. Check commit.interval.ms and processing time per batch.
• Kafka Connect exactly-once source support (exactly.once.source.support=enabled on the worker) uses transactional producers per task. Fencing appears after rebalances if zombie tasks keep running; check worker logs for both instances of the task.

#Step 6: monitor

#Prevent it next time

• Unique, stable transactional.id per producer instance; never shared between replicas.
• Short transactions with a sensible transaction.timeout.ms.
• Keep brokers and clients current; many hanging-transaction bugs were fixed in Kafka 3.x and the KIP-890 transaction protocol hardening in later versions.
• Run find-hanging regularly on clusters that use transactions.