Kafka producer TimeoutException: expiring records - find whether it's the client, network or broker

#Symptom

Producers log something like this, and send callbacks fail:

org.apache.kafka.common.errors.TimeoutException: Expiring 42 record(s) for orders-3:
120000 ms has passed since batch creation

A close relative is:

Topic orders not present in metadata after 60000 ms.

These two have different causes. The first means a batch sat waiting too long for delivery. The second means the client couldn't get metadata, usually because of bootstrap, listener, authorization or topic-name problems, and it's bounded by max.block.ms. If you have the second one, read advertised listeners and authorization errors first.

#The timing model

delivery.timeout.ms (default 120 seconds) limits the time from send() to the final result. It covers:

Find which stage has used up the time before changing any of them. Raising delivery.timeout.ms alone only hides the problem and makes the backlog bigger.

#Step 1: read the producer metrics

These come from the JMX group kafka.producer:type=producer-metrics,client-id=*:

Print them from a Java client:

producer.metrics().forEach((name, metric) -> {
    if (name.group().equals("producer-metrics") &&
        List.of("record-queue-time-avg", "request-latency-avg", "buffer-available-bytes",
                "record-retry-rate", "record-error-rate", "batch-size-avg").contains(name.name())) {
        System.out.println(name.name() + " = " + metric.metricValue());
    }
});

#Step 2: is it one partition or all of them?

kafka-topics.sh --bootstrap-server "$BOOTSTRAP" --describe --topic orders
kafka-topics.sh --bootstrap-server "$BOOTSTRAP" --describe --under-replicated-partitions

The partition in the error message tells you a lot. Compare errors across partitions and find their leaders:

#Step 3: check broker health

Key broker metrics:

Look at the disks too:

iostat -xm 5 3
df -h /var/lib/kafka

A high RemoteTimeMs with acks=all means the leader is waiting on followers, which sends you looking at a slow disk or network on another broker. Also check under-replicated partitions.

Also check for quota throttling, which delays responses by design:

kafka-configs.sh --bootstrap-server "$BOOTSTRAP" --describe --entity-type clients

The client metric produce-throttle-time-avg shows whether it's being applied.

#Step 4: baseline the path

Compare your app with a clean producer, using the same settings, from the same host:

kafka-producer-perf-test.sh --topic perf-test \
  --num-records 200000 --record-size 1024 --throughput -1 \
  --producer-props bootstrap.servers="$BOOTSTRAP" acks=all linger.ms=10 \
  batch.size=131072 compression.type=lz4

Use a dedicated test topic. If this is fast and your app isn't, look at the application: blocking calls in a callback, too many producers, or very large records.

#Step 5: tune by cause

A reasonable starting point:

acks=all
enable.idempotence=true
compression.type=lz4
linger.ms=20
batch.size=131072
buffer.memory=134217728
request.timeout.ms=30000
delivery.timeout.ms=120000
max.block.ms=60000

The rule delivery.timeout.ms >= linger.ms + request.timeout.ms must hold, or the producer won't start. For more, see throughput tuning.

#Handle the timeout correctly in the app

Warning

A TimeoutException doesn't prove the record failed. The broker may have written it after the client gave up. Blindly sending it again from your application can create a duplicate, because idempotence only protects retries inside the same producer session.

producer.send(record, (metadata, exception) -> {
    if (exception == null) return;
    if (exception instanceof RetriableException) {
        // Delivery state unknown: park it for a checked replay
        deadLetter.park(record, exception);
    } else {
        alert(record, exception);
    }
});

• Always pass a callback. Without one, or without checking the returned Future, failures vanish.
• Make consumers idempotent, using a business key, for records you replay.
• Call producer.flush() and close() during shutdown, so buffered records aren't lost.

Caution

Don't set acks=1 or acks=0 to make the timeouts go away. It stops the errors by giving up the durability guarantee, and you can lose data on a broker failure.

#Prevent it next time

• Export producer metrics, and alert on record-error-rate above 0 and on rising queue time.
• Load-test at expected peak traffic before launch, with a realistic record size.
• Check RequestHandlerAvgIdlePercent during capacity reviews.
• Related: min.insync.replicas errors and broker disk full.