Home / Data Platform / Streaming & CDC

Data PlatformChecklistAdvanced

Kafka 4.x upgrade readiness - KRaft first, clients 2.1+, Java 17 brokers and removed configs

Kafka 4.0 removed ZooKeeper and old protocol versions, raised Java minimums and removed several configs. Before you upgrade, confirm KRaft mode on 3.3+ metadata, find clients older than 2.1, move brokers to Java 17, and clean configs that no longer exist.

Deepak Bhardwaj3 min read

Applies toApache Kafka 3.x to 4.x (self-managed); Kafka Connect, Kafka Streams and Java/non-Java clients; also relevant when a managed Kafka service moves you to 4.x

The rules from the 4.0 upgrade notes

Requirement Detail
KRaft only "Apache Kafka 4.0 only supports KRaft mode - ZooKeeper mode has been removed." ZooKeeper clusters must be migrated to KRaft before upgrading to 4.0
Metadata version Brokers upgrading to 4.0 must run KRaft with software and metadata version at least 3.3.x. For KRaft clusters older than 3.3, upgrade to 3.9.x first
Client floor Clients (including Streams and Connect) must be 2.1 or newer before upgrading to 4.0; brokers must be 2.1+ before upgrading Java clients to 4.x
Java Clients and Streams need Java 11+; brokers, Connect and tools need Java 17
Removed configs For example log.message.format.version, message.format.version, offsets.commit.required.acks, delegation.token.master.key, auto.include.jmx.reporter, log.message.timestamp.difference.max.ms

If you're still on ZooKeeper, start with the migration itself: see ZooKeeper to KRaft migration.

1. Confirm KRaft and metadata version

bin/kafka-metadata-quorum.sh --bootstrap-server broker1:9092 describe --status
bin/kafka-features.sh --bootstrap-server broker1:9092 describe

describe --status only works on KRaft. In the features output, check metadata.version. If it's below 3.3, plan an intermediate upgrade to 3.9.x and finalise the metadata version there:

bin/kafka-features.sh --bootstrap-server broker1:9092 upgrade --release-version 3.9

2. Find old clients

Old clients are the most common surprise, because they live in other teams' applications. Kafka 4.0 removed old protocol API versions, so clients older than 2.1 stop working.

Ways to find them:

  • Broker request metrics: brokers expose request metrics per API and version (kafka.network:type=RequestMetrics,name=RequestsPerSec,request=...,version=...). Look for old versions of Produce, Fetch and Metadata.
  • Client software name and version: newer clients send them; check broker logs or metrics where available.
  • Ask: list consumer groups and map them to owning applications.
bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --list

Non-Java clients (librdkafka-based Python, Go, .NET, Node clients) have their own versioning. Check each library's release notes for Kafka 4 compatibility.

3. Java

  • Brokers, Connect workers and admin tools: Java 17 runtime.
  • Applications using Java clients or Streams: Java 11 minimum.
  • Connect plugins (connectors, converters, SMTs) run inside the Connect JVM, so they must work on Java 17 too. Test each one.

4. Clean configs

grep -nE "log\.message\.format\.version|^message\.format\.version|offsets\.commit\.required\.acks|delegation\.token\.master\.key|auto\.include\.jmx\.reporter|log\.message\.timestamp\.difference\.max\.ms" \
  config/server.properties config/*.properties
  • Replace delegation.token.master.key with delegation.token.secret.key.
  • Replace log.message.timestamp.difference.max.ms with log.message.timestamp.before.max.ms and log.message.timestamp.after.max.ms.
  • Check topic-level overrides too (kafka-configs.sh --describe --entity-type topics --all).

This list isn't complete. Read "Notable changes in 4.0.0" (and later versions you're jumping to) in full.

5. Upgrade order

  1. Upgrade any clients below 2.1.
  2. Brokers on KRaft with metadata version >= 3.3 (3.9 recommended).
  3. Java 17 on broker hosts and Connect.
  4. Rolling broker upgrade to 4.x, one broker at a time, checking under-replicated partitions between each.
  5. When everything is stable, finalise the new metadata version with kafka-features.sh upgrade. Finalising can't easily be undone, so wait until you're sure.

Checklist

  • Cluster in KRaft mode with metadata version 3.3+ (ideally 3.9)
  • No clients older than 2.1; non-Java libraries checked
  • Brokers, Connect and tools on Java 17; clients on Java 11+
  • Connect plugins tested on Java 17
  • Removed configs cleaned from broker and topic settings
  • Rolling upgrade plan with per-broker health checks
  • Metadata version finalised only after a soak period

Sources

Related fixes