Kafka clients connect, then time out: fixing advertised.listeners and bootstrap metadata

#Symptoms

• A client reaches the bootstrap port, then fails with one of these messages. Wording varies by client version:
  • Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Node may not be available.
  • Topic orders not present in metadata after 60000 ms.
  • Timed out waiting for a node assignment.
  • Error connecting to node kafka-1:9092 (id: 1 rack: null) with UnknownHostException: kafka-1
• Clients inside the VNet, VPC or Docker network work. Clients outside it do not.

Kafka connections happen in two steps. The client uses bootstrap.servers only to fetch metadata. It then connects directly to the broker addresses in that metadata. If those addresses are wrong or unreachable from the client, the first step passes and the second one fails.

In client logs, a negative node ID such as node -1 is the bootstrap connection. A positive ID is a broker address taken from metadata. That one detail tells you which step failed.

#Checks

#1. Read what the cluster advertises

Run this from the failing client's network, using the same bootstrap address and port as the client.

# kcat prints the brokers from the metadata response
kcat -b kafka-bootstrap.example.com:9094 -L -F client.conf | sed -n '1,10p'

# Kafka CLI alternative: each broker line starts with its advertised host:port
kafka-broker-api-versions.sh \
  --bootstrap-server kafka-bootstrap.example.com:9094 \
  --command-config client.properties | grep 'id:'

Look for loopback addresses, private IPs, container names and short hostnames. Those are fine only if this client can resolve and reach them.

#2. Test every advertised address from the client host

for b in kafka-1.example.com:9094 kafka-2.example.com:9094 kafka-3.example.com:9094; do
  host="${b%:*}"; port="${b#*:}"
  getent hosts "$host" || echo "DNS FAILED: $host"
  nc -vz -w 3 "$host" "$port"
done

One failing broker produces partial failures: only partitions led by that broker time out. A client may seem healthy until it needs that broker.

#3. Read the broker configuration, including dynamic overrides

# Static config (path varies by install)
grep -E '^(listeners|advertised\.listeners|listener\.security\.protocol\.map|inter\.broker\.listener\.name|controller\.listener\.names)=' \
  /opt/kafka/config/server.properties

# Dynamic overrides for broker 1
kafka-configs.sh --bootstrap-server kafka-1.internal.example.com:9092 \
  --command-config admin.properties \
  --entity-type brokers --entity-name 1 --describe

# Effective values, whatever their source
kafka-configs.sh --bootstrap-server kafka-1.internal.example.com:9092 \
  --command-config admin.properties \
  --entity-type brokers --entity-name 1 --describe --all | grep -i listener

Precedence is dynamic per-broker config, then dynamic cluster-wide default, then server.properties, then the Kafka default. An old override will beat the file you just edited.

#Symptom, cause and fix

The listener a client connects to decides which addresses it gets back. A client that bootstraps on the EXTERNAL port receives the EXTERNAL addresses of the brokers, even if it could also reach the internal ones.

#The fix: one listener per network

Bind with listeners and tell clients where to go with advertised.listeners. This is broker 1 in KRaft mode:

process.roles=broker
node.id=1

listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9094
advertised.listeners=INTERNAL://kafka-1.internal.example.com:9092,EXTERNAL://kafka-1.example.com:9094

listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL,CONTROLLER:SSL
inter.broker.listener.name=INTERNAL
controller.listener.names=CONTROLLER

The broker-only example deliberately leaves the controller listener out of listeners, but it still needs the name and its security settings. On combined broker,controller nodes, the controller listener must be in listeners. Add your controller quorum setting, plus keystore and authentication settings for each protocol. The controller listener cannot be the same as the inter-broker listener.

Warning

The INTERNAL address is also how other brokers replicate from this one. If you advertise a wrong address there, followers fall behind and partitions go under-replicated. Change one broker first and check it before moving on.

#Roll it out

Update the file for each broker with that broker's own hostnames, then restart one broker at a time.

# After each restart, and before touching the next broker. Expect no output.
kafka-topics.sh --bootstrap-server kafka-1.internal.example.com:9092 \
  --command-config admin.properties \
  --describe --under-replicated-partitions

Kafka 4.0's documentation is inconsistent about this setting. Its configuration table lists advertised.listeners as read-only, while its section on adding and removing listeners says it can change per broker without a restart. Use the rolling restart unless you have tested a dynamic update on your exact version.

If check 3 found a stale dynamic override, remove it so the file takes effect:

kafka-configs.sh --bootstrap-server kafka-1.internal.example.com:9092 \
  --command-config admin.properties \
  --entity-type brokers --entity-name 1 --alter --delete-config advertised.listeners

Caution

Changing inter.broker.listener.name needs its own careful rolling restart, and a mistake there can break replication across the cluster. Add the new listener to every broker first. Then switch the inter-broker listener in a separate roll.

#Docker

A broker in a container needs one address for other containers and another for the host. With the apache/kafka image, environment variables map to the same settings:

services:
  kafka:
    image: apache/kafka:latest   # pin a version in anything you keep
    ports:
      - "29092:29092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: INTERNAL://0.0.0.0:9092,HOST://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,HOST:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Containers use kafka:9092. Applications on the host use localhost:29092. This is a dev setup with no TLS or authentication.

#Load balancers and Kubernetes

A single load balancer address works for bootstrap, but it cannot replace broker addresses. A client must reach the specific broker that leads each partition. If every broker advertises the same address and port, requests land on the wrong broker and fail.

Give each broker its own external name or port, such as one load balancer per broker or one port per broker. On Kubernetes, use a per-broker external listener. Operators such as Strimzi can generate one.

#Confirm the fix

Run these from each client network, using a test topic that already exists:

kcat -b kafka-bootstrap.example.com:9094 -F client.conf -L | sed -n '1,10p'
echo "listener-test $(date -u +%FT%TZ)" | kcat -P -b kafka-bootstrap.example.com:9094 -F client.conf -t ops-smoke-test
kcat -C -b kafka-bootstrap.example.com:9094 -F client.conf -t ops-smoke-test -o -1 -c 1 -e

The produce and consume test proves the client reached the partition leader.

#Prevent it next time

• Generate each broker's listeners and advertised.listeners from a template, so hostnames and broker IDs can't be copied wrongly.
• Keep one named listener per network path, and document which clients use which one.
• Use stable DNS names, not IPs that change when a VM or pod is replaced.
• Include every advertised name in the certificate SAN before you roll it out.
• Run the smoke test from each client network after every broker, firewall or DNS change.