Kafka SSL handshake failed: expired certificates, hostname verification and truststore problems

#Symptoms

• Producers and consumers fail to connect and the client log shows SSL handshake failed followed by javax.net.ssl.SSLHandshakeException.
• Broker logs show Failed authentication with /10.0.4.17 (channelId=...) (SSL handshake failed).
• A broker that just restarted cannot rejoin: inter-broker or controller traffic fails over the SSL listener and partitions go under-replicated.
• Everything worked until a specific date, which usually means a certificate expired.

The handshake error is generic. The Caused by: line underneath it tells you which side rejected which certificate. Get that line before changing anything.

#Quick checks

#1. Look at the certificate the broker actually serves

Do this from the client host, so you test the same DNS name, network path and listener the client uses.

openssl s_client -connect kafka-1.prod.example.com:9093 \
  -servername kafka-1.prod.example.com -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

Check three things:

• notAfter is in the future.
• The host name in bootstrap.servers and every name in the broker's advertised.listeners appears in subjectAltName. Clients connect to the advertised names after the first metadata call, not only to the bootstrap name.
• The issuer is a CA that the client truststore contains.

If s_client prints Verify return code: 21 (unable to verify the first certificate), the broker is not sending its intermediate CA. Add the full chain to the broker keystore.

#2. Inspect the keystore and truststore files

# Broker keystore: alias, validity and chain length
keytool -list -v -keystore /etc/kafka/ssl/kafka.server.keystore.jks \
  -storepass "$KS_PASS" | grep -E 'Alias name|Valid from|Owner|Issuer|Certificate chain length'

# Client truststore: which CAs does the client trust?
keytool -list -v -keystore /etc/kafka/ssl/client.truststore.jks \
  -storepass "$TS_PASS" | grep -E 'Alias name|Owner|Valid from'

For PEM files (ssl.keystore.type=PEM, Kafka 2.7+):

openssl x509 -in /etc/kafka/ssl/broker.crt -noout -dates -subject -ext subjectAltName

#3. Reproduce with the Kafka CLI

# client-ssl.properties
security.protocol=SSL
ssl.truststore.location=/etc/kafka/ssl/client.truststore.jks
ssl.truststore.password=changeit
# only when the broker sets ssl.client.auth=required
ssl.keystore.location=/etc/kafka/ssl/client.keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit

kafka-broker-api-versions.sh --bootstrap-server kafka-1.prod.example.com:9093 \
  --command-config client-ssl.properties

Use SASL_SSL and add your SASL settings if the listener uses SASL. For full handshake detail, turn on JSSE debug for one run only, because it is very verbose:

KAFKA_OPTS="-Djavax.net.debug=ssl:handshake" \
  kafka-broker-api-versions.sh --bootstrap-server kafka-1.prod.example.com:9093 \
  --command-config client-ssl.properties 2> handshake-debug.log

#Error to cause to fix

Warning

Setting ssl.endpoint.identification.algorithm= (empty) turns off hostname verification. It makes the error go away and leaves clients open to man-in-the-middle attacks. Use it only as a short, documented workaround while a correctly named certificate is issued.

#Fix: rotate a broker certificate without restarting the cluster

Kafka supports dynamic updates of SSL keystores per listener (KIP-226). The new keystore must contain a certificate for the same distinguished name and SANs that clients already accept, or the update is rejected.

1. Build the new keystore with the key, the new certificate and the full chain. Write it to a new file name so the change is unambiguous:

openssl pkcs12 -export -in broker-2026.crt -inkey broker-2026.key \
  -certfile intermediate-and-root.crt -name kafka-1 \
  -out /etc/kafka/ssl/kafka.server.keystore.2026.p12 -passout pass:"$KS_PASS"

2. Point the listener at it, one broker at a time. The listener name is lowercase in the config key:

kafka-configs.sh --bootstrap-server kafka-1.prod.example.com:9093 \
  --command-config admin-ssl.properties \
  --entity-type brokers --entity-name 1 --alter \
  --add-config 'listener.name.ssl.ssl.keystore.location=/etc/kafka/ssl/kafka.server.keystore.2026.p12,listener.name.ssl.ssl.keystore.type=PKCS12,listener.name.ssl.ssl.keystore.password='"$KS_PASS"

3. Re-run the openssl s_client check against that broker and confirm the new notAfter.
4. Update server.properties to match, so a restart doesn't revert to the old file.

If the broker is already down because of an expired certificate, the dynamic path isn't available on that node. Replace the file referenced in server.properties and restart that broker only.

Caution

If every broker certificate expired at the same moment, inter-broker replication and the KRaft controller quorum may also be failing. Fix the controllers first, then brokers, and restart one node at a time. Restarting everything at once with a still-broken keystore turns a client outage into a full cluster outage.

#Adding a new CA

When you move to a new CA, add the new CA to every truststore (brokers and clients) first, roll that out, and only then reissue certificates from the new CA. Reversing the order breaks every client that hasn't received the new truststore.

keytool -importcert -noprompt -alias new-root-2026 -file new-root.crt \
  -keystore /etc/kafka/ssl/client.truststore.jks -storepass "$TS_PASS"

Truststore changes can also be applied dynamically on brokers with listener.name.<listener>.ssl.truststore.location. Clients pick them up on restart.

#Prevent it next time

• Monitor expiry from outside the broker. A scheduled check works anywhere:

for b in kafka-1 kafka-2 kafka-3; do
  end=$(openssl s_client -connect $b.prod.example.com:9093 -servername $b.prod.example.com </dev/null 2>/dev/null \
        | openssl x509 -noout -enddate | cut -d= -f2)
  echo "$b $end"
  openssl s_client -connect $b.prod.example.com:9093 </dev/null 2>/dev/null \
    | openssl x509 -noout -checkend $((30*24*3600)) >/dev/null || echo "ALERT: $b expires within 30 days"
done

• Put every advertised host name, the bootstrap alias and any load balancer name in the SAN list from the start.
• Keep keystores and passwords in a secrets manager and template them, so rotation is a pipeline run, not a manual edit on each broker.
• Rehearse a dynamic keystore update in a non-production cluster so the commands and listener names are known before you need them.
• Record certificate owners and expiry dates for client keystores too. With mutual TLS, an expired client certificate is a producer outage the broker team often never sees.