Connecting Kafka clients to Azure Event Hubs: working config and fixes for timeouts, auth and compression errors

#The goal

You have a Kafka application (Java, Python confluent-kafka, .NET, Go, Kafka Connect, Spark) and you want it to produce to or consume from Azure Event Hubs without code changes. Event Hubs exposes a Kafka-compatible endpoint, so this is mostly configuration. Most incidents come from Kafka client defaults that don't suit Event Hubs: idle connections, request timeouts, request size and compression.

The mapping you need in your head:

#Check the namespace first

The Kafka endpoint works on the Standard, Premium and Dedicated tiers. Basic does not support it. On a Basic namespace the client usually fails with TopicAuthorizationException, which sends people off to look at permissions that are actually fine.

# Tier and Kafka support for the namespace
az eventhubs namespace show -g rg-stream -n ehns-prod \
  --query "{sku:sku.name, kafka:kafkaEnabled, publicAccess:publicNetworkAccess}" -o table

# The "topic" must already exist as an event hub
az eventhubs eventhub list -g rg-stream --namespace-name ehns-prod \
  --query "[].{name:name, partitions:partitionCount, status:status}" -o table

Kafka clients can't create topics on Event Hubs through the AdminClient. Create the event hub first, and turn off auto.create.topics.enable-style behaviour in frameworks that try to create topics on startup:

az eventhubs eventhub create -g rg-stream --namespace-name ehns-prod \
  -n orders --partition-count 8

Warning

You can't reduce an event hub's partition count after creation, and on Standard you can't increase it either. Choose the partition count as carefully as you would for a Kafka topic.

Network: the client must reach port 9093 on the namespace FQDN. If the namespace uses a private endpoint, the FQDN must resolve to the private IP from the client host.

nslookup ehns-prod.servicebus.windows.net
nc -vz ehns-prod.servicebus.windows.net 9093

#Authentication options

#Connection string (SASL PLAIN)

The username is the literal string $ConnectionString. The password is the full connection string for a shared access policy on the namespace or event hub.

az eventhubs namespace authorization-rule keys list -g rg-stream \
  --namespace-name ehns-prod -n app-orders-send --query primaryConnectionString -o tsv

Java client (client.properties):

bootstrap.servers=ehns-prod.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://ehns-prod.servicebus.windows.net/;SharedAccessKeyName=app-orders-send;SharedAccessKey=<key>";

Python (confluent-kafka, librdkafka):

from confluent_kafka import Producer

conf = {
    "bootstrap.servers": "ehns-prod.servicebus.windows.net:9093",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "PLAIN",
    "sasl.username": "$ConnectionString",
    "sasl.password": conn_str,          # from Key Vault, not source code
    "request.timeout.ms": 60000,
    "metadata.max.age.ms": 180000,
    "socket.keepalive.enable": True,
    "compression.codec": "gzip",
}
p = Producer(conf)
p.produce("orders", key="42", value=b'{"id":42}')
p.flush(30)

Two things that catch people out:

• Generated SAS tokens aren't accepted on the Kafka endpoint. Use the connection string with the key, or Entra ID.
• When you regenerate a SAS key, existing Kafka connections aren't dropped. The application keeps working until it reconnects, then fails. Rotate the key you're not using, update the clients, and only then regenerate the other one.

#Microsoft Entra ID (SASL OAUTHBEARER)

For production, use OAuth with a managed identity or service principal and Azure RBAC (Azure Event Hubs Data Sender / Azure Event Hubs Data Receiver). The client config uses sasl.mechanism=OAUTHBEARER plus a login callback handler that fetches the token. Microsoft publishes Java and other samples in the Azure/azure-event-hubs-for-kafka GitHub repo under tutorials/oauth.

az role assignment create --assignee <principal-id> \
  --role "Azure Event Hubs Data Receiver" \
  --scope $(az eventhubs eventhub show -g rg-stream --namespace-name ehns-prod -n orders --query id -o tsv)

#Client settings that differ from Kafka defaults

Event Hubs closes connections that are idle for 240 seconds. It also enforces a request size limit and a minimum request timeout. These are the values Microsoft recommends:

Defaults change between client releases. Check the version you run, and check the current Microsoft page for these values before you change production.

Tip

Idempotent producers are supported, and enable.idempotence=true is the default in recent Java clients. It needs acks=all, a high retries value and request.timeout.ms of at least 60000, so the timeout change above is required either way.

#Symptom → cause → fix

#Verify it end to end

Use the standard Kafka console tools with the same client.properties:

kafka-console-producer.sh --bootstrap-server ehns-prod.servicebus.windows.net:9093 \
  --producer.config client.properties --topic orders

kafka-console-consumer.sh --bootstrap-server ehns-prod.servicebus.windows.net:9093 \
  --consumer.config client.properties --topic orders --group ops-smoke-test --from-beginning

Then confirm on the Azure side that messages arrived and nothing was throttled:

az monitor metrics list \
  --resource $(az eventhubs namespace show -g rg-stream -n ehns-prod --query id -o tsv) \
  --metric IncomingMessages OutgoingMessages ThrottledRequests ServerErrors \
  --interval PT5M --aggregation Total -o table

Kafka throttling doesn't always appear in ThrottledRequests. Also check throttle_time_ms in the client metrics.

#Prevent it next time

• Keep one shared, reviewed client config per language with the Event Hubs settings above, and have teams import it instead of writing their own.
• Create event hubs through IaC (az or Bicep) with the partition count agreed in review. Disable topic auto-creation in Kafka Connect and frameworks.
• Use Entra ID with managed identities in production. If you must use SAS, keep keys in Key Vault and rotate them using the two-key method.
• Alert on ServerErrors, ThrottledRequests and consumer lag. Put the 9093 connectivity check in your deployment smoke tests.
• Before you rely on transactions, Kafka Streams or compression, check which tier supports them now. Microsoft's own pages have disagreed on this, and support differs between Standard and Premium/Dedicated.