Kafka RecordTooLargeException - raising message size limits on producers, brokers, topics and consumers

A producer or a Kafka Connect source task starts failing with RecordTooLargeException. Most messages still get through. A few large ones fail every time. Raising the size limit takes four settings in a fixed order. Change only one and the failure moves to a different component.

#Identify which limit was hit

Read the exact error. The wording tells you which side rejected the record.

The defaults are all about 1 MB. Check your version's documentation for exact values, because names and defaults change between releases.

#How the limits fit together

• Producer max.request.size – the largest request the producer sends. It also caps the size of a single record.
• Topic max.message.bytes – the largest record batch the broker accepts for that topic, after compression. It overrides the broker default for that topic.
• Broker message.max.bytes – the cluster-wide default for topics that don't set max.message.bytes.
• Broker replica.fetch.max.bytes – the fetch size followers use to replicate. Since Kafka 0.10.1 (KIP-74) this is not an absolute maximum. A batch bigger than this is still fetched if it's the first one in the partition. Keep it at least as large as the biggest batch anyway, so replication doesn't slow down.
• Consumer max.partition.fetch.bytes and fetch.max.bytes – fetch sizes. As with replication, they are not absolute limits in modern clients. A single oversized batch is still returned. Raise them anyway to keep throughput normal.

Compression matters. The producer checks the serialized size of a record before compression. The broker checks the compressed batch. A 3 MB JSON payload might compress to 400 KB and pass the broker, but it still fails at the producer unless max.request.size is at least 3 MB.

#Step 1: measure the real message size

Don't guess a new limit. Find out how big the failing records are. Log the serialized size in the producer, or check the error message, which gives the size in bytes. For an existing topic, look at the largest batches on disk:

kafka-dump-log.sh --files /var/lib/kafka/data/orders-0/00000000000000000000.log \
  --print-data-log | grep -o 'size: [0-9]*' | sort -t' ' -k2 -n | tail -5

Set the new limit to the largest real message plus headroom, for example 2× the largest. Don't use something like 100 MB "to be safe".

Warning

Large limits have a cost. Brokers allocate memory per fetch and per request, and every partition a consumer reads can return a batch up to the fetch size. Very large messages increase GC pressure and replication latency, and can push consumers past max.poll.interval.ms. If messages are routinely over a few MB, see "Shrink the messages instead" below.

#Step 2: raise the topic limit

Change only the topic that needs it, not the whole cluster:

kafka-configs.sh --bootstrap-server broker1:9092 --command-config admin.properties \
  --entity-type topics --entity-name orders \
  --alter --add-config max.message.bytes=4194304

kafka-configs.sh --bootstrap-server broker1:9092 --command-config admin.properties \
  --entity-type topics --entity-name orders --describe

Topic config changes apply without a broker restart.

#Step 3: check replication fetch size on the brokers

Check replica.fetch.max.bytes on the brokers:

kafka-configs.sh --bootstrap-server broker1:9092 --command-config admin.properties \
  --entity-type brokers --entity-name 1 --describe --all | grep -E 'replica.fetch.max.bytes|message.max.bytes'

If it is smaller than the new topic limit, raise it cluster-wide. It is dynamically updatable as a cluster default in current versions. Check the "Dynamic Update Mode" for your version in the broker config docs.

kafka-configs.sh --bootstrap-server broker1:9092 --command-config admin.properties \
  --entity-type brokers --entity-default \
  --alter --add-config replica.fetch.max.bytes=4194304

Watch under-replicated partitions for the topic after the first large messages arrive:

kafka-topics.sh --bootstrap-server broker1:9092 --command-config admin.properties \
  --describe --topic orders --under-replicated-partitions

On Amazon MSK, put message.max.bytes and replica.fetch.max.bytes in a custom cluster configuration revision and apply it with aws kafka update-cluster-configuration. Topic-level max.message.bytes still works through kafka-configs.sh.

#Step 4: raise the producer limit

# producer.properties
max.request.size=4194304
# make sure buffer.memory (default ~32 MB) is comfortably above max.request.size
buffer.memory=67108864
compression.type=zstd

batch.size doesn't have to grow. A record bigger than batch.size is simply sent in its own batch.

#Kafka Connect source connectors

Connect workers create the producers, so set the override per connector:

{
  "producer.override.max.request.size": "4194304"
}

curl -s -X PUT -H "Content-Type: application/json" \
  --data @orders-source-config.json \
  http://connect:8083/connectors/orders-source/config

This only works if the worker allows overrides through connector.client.config.override.policy. The default depends on your version and distribution, so set it explicitly to All or a restrictive policy. If overrides are blocked, set producer.max.request.size in the worker config instead. Changing the worker config needs a rolling restart of the workers.

#Step 5: raise the consumer fetch sizes

# consumer.properties
max.partition.fetch.bytes=4194304
fetch.max.bytes=52428800

For sink connectors, use consumer.override.max.partition.fetch.bytes. Mirror the change on any MirrorMaker 2 or replication tooling that reads the topic, and make sure the target cluster's topic has the same max.message.bytes. Otherwise replication fails there.

Caution

Roll out in this order: topic, then brokers, then consumers, then producers last. If producers go first, they write messages that downstream components may handle badly, and a mirrored cluster can reject them outright.

#Managed services with fixed limits

On some managed services the maximum message size can't be changed, or depends on the tier. This includes the Azure Event Hubs Kafka endpoint and some Confluent Cloud cluster types. Check the current limits for your tier. If you're over the limit, raising client settings won't help. Shrink the messages.

#Shrink the messages instead

Often the better fix is to stop sending huge records:

• Compress. Use compression.type=zstd or lz4 on the producer. This helps against broker limits, not the producer's pre-compression check.
• Use the claim-check pattern. Write the payload to S3, ADLS or OCI Object Storage and send a small event with the object URI and a checksum.
• Remove bloat. Drop embedded base64 files and repeated nested arrays. Switch JSON to Avro or Protobuf with Schema Registry.
• Split logically. Send one event per order line instead of one event per order with 10,000 lines. Keep the key the same so ordering per entity holds.

#Prevent it next time

• Put a size check in producer code that logs and routes oversized records to a dead-letter topic instead of failing the whole batch. For Connect sinks, use errors.tolerance=all with errors.deadletterqueue.topic.name.
• Keep max.message.bytes, replica.fetch.max.bytes and the client settings in the same infrastructure-as-code change, so they can't drift apart.
• Alert on producer record-error-rate and on Connect task failures. A single bad record type shows up there first.
• Agree on a maximum message size in the data contract with the producing team.