Applies toApache Kafka 3.x/4.x clients (Java and librdkafka-based), Amazon MSK, Confluent, Azure Event Hubs Kafka endpoint
Measure first
| Signal | Where |
|---|---|
Producer record-send-rate, batch-size-avg, records-per-request-avg, request-latency-avg, compression-rate-avg | Producer JMX / client metrics |
Consumer records-lag-max, fetch-rate, records-per-request-avg, poll interval | Consumer metrics |
| Broker request queue time, network/IO thread idle %, under-replicated partitions | Broker JMX |
Small batch-size-avg and high request rates mean the producer sends many tiny requests, the most common inefficiency.
Producer
| Setting | Effect | Starting point for throughput |
|---|---|---|
batch.size | Max bytes per partition batch | 64–256 KB |
linger.ms | Wait to fill batches | 5–50 ms (trade a little latency for much better batching) |
compression.type | Fewer bytes on network and disk | lz4 or zstd |
acks | Durability | all for important data |
enable.idempotence | No duplicates on retry | true (default in recent clients) |
max.in.flight.requests.per.connection | Pipelining | ≤ 5 with idempotence |
buffer.memory | Client-side buffer | Increase if buffer-available-bytes hits zero |
acks=all
enable.idempotence=true
compression.type=zstd
batch.size=131072
linger.ms=20
Defaults change between versions (e.g. linger.ms). Set them explicitly so behaviour doesn't change on upgrade.
Partitioning: a good key spreads load; a hot key sends everything to one partition. Watch per-partition bytes-in.
Consumer
| Setting | Effect |
|---|---|
fetch.min.bytes / fetch.max.wait.ms | Fewer, larger fetches (e.g. 64 KB / 100–500 ms) |
max.partition.fetch.bytes, fetch.max.bytes | Allow larger fetches for big messages or high throughput |
max.poll.records | Records per poll. Size so processing finishes within max.poll.interval.ms |
max.poll.interval.ms | Max time between polls before the consumer is kicked out |
partition.assignment.strategy | Cooperative sticky assignor to reduce rebalance pauses |
group.protocol=consumer | New rebalance protocol (KIP-848) where supported. Check broker/client versions |
Processing, not fetching, is usually the bottleneck:
- Batch writes to the downstream system (databases, APIs).
- Process in parallel inside the consumer carefully (ordering per partition matters).
- Scale consumers up to the number of partitions; more consumers than partitions sit idle.
Broker-side checks (after clients)
num.network.threads,num.io.threadsvs idle percentages.- Disk throughput and page cache: consumers reading old data (catch-up) cause disk reads.
- Partition count: enough for consumer parallelism, not thousands per broker without reason.
- Replication:
replica.fetch.max.bytes, follower lag.
A practical tuning loop
- Baseline with
kafka-producer-perf-test.sh/kafka-consumer-perf-test.shusing real message sizes. - Change one setting, re-test, record throughput and p99 latency.
- Apply to the application, watch production metrics for a week.
kafka-producer-perf-test.sh --topic perf-test --num-records 2000000 --record-size 1024 --throughput -1 \
--producer-props bootstrap.servers=broker:9092 acks=all compression.type=zstd batch.size=131072 linger.ms=20
Warning
Don't trade durability for speed silently:
acks=1or disabling idempotence can lose or duplicate data during broker failures. If you change them, document why and for which topics.
Checklist
- Producer and consumer metrics collected
- Batching, linger and compression tuned
- Durability settings explicit (acks=all, idempotence)
- Consumer fetch/poll settings sized to processing
- Consumers ≤ partitions; hot keys checked
- Broker checks after client tuning; perf tests recorded