Kafka ACL errors: fixing TopicAuthorizationException, GroupAuthorizationException and other authorization failures

#Symptoms

• A producer fails with TopicAuthorizationException: Not authorized to access topics: [orders].
• A consumer connects, then fails with GroupAuthorizationException: Not authorized to access group: orders-svc.
• An idempotent or transactional producer fails with ClusterAuthorizationException or TransactionalIdAuthorizationException.
• Everything worked until someone enabled an authorizer or set allow.everyone.if.no.acl.found=false.

#How Kafka decides

With an authorizer enabled, every request is checked against ACLs: principal + operation + resource (+ host). If no ACL allows it, it is denied, unless the resource has no ACLs at all and allow.everyone.if.no.acl.found=true. Principals listed in super.users bypass ACLs.

The authorizer class depends on the cluster mode:

#Step 1: find the principal the broker sees

Most ACL problems are a principal-name mismatch. The ACL says User:orders-app; the broker sees something else.

The broker's authorizer log shows exactly what was denied. Denials are logged at INFO by the kafka.authorizer.logger logger (allows only at DEBUG). The log is often a separate file such as kafka-authorizer.log:

grep -i "denied" /var/log/kafka/kafka-authorizer.log | tail -20

A line looks like:

Principal = User:CN=orders-app,OU=Apps,O=Example,C=GB is Denied operation = READ
from host = 10.20.1.45 on resource = Topic:LITERAL:orders for request = Fetch

That line tells you the principal, operation, resource type, pattern type and topic name. Use it to write the ACL.

#Step 2: list existing ACLs

Admin commands need a client config with admin credentials:

# admin.properties
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="...";

kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties --list

kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
  --list --topic orders

kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
  --list --principal User:orders-app

#Step 3: add the minimal ACLs

#Consumer

A consumer needs Read on the topic and Read on its consumer group:

kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
  --add --allow-principal User:orders-app \
  --operation Read --operation Describe --topic orders

kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
  --add --allow-principal User:orders-app \
  --operation Read --group orders-svc

Or use the convenience flag, which adds both:

kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
  --add --allow-principal User:orders-app --consumer --topic orders --group orders-svc

#Producer

kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
  --add --allow-principal User:billing-app --producer --topic invoices

--producer grants Write, Describe and Create on the topic.

#Transactional or idempotent producer

kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
  --add --allow-principal User:billing-app --producer --topic invoices \
  --transactional-id billing-tx --idempotent

On older brokers, idempotent producers need IdempotentWrite on the cluster; newer versions accept Write on the topic. If you see ClusterAuthorizationException from a producer with enable.idempotence=true (the default in recent clients), adding --idempotent fixes it.

#Prefixed ACLs for many topics

kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
  --add --allow-principal User:analytics \
  --operation Read --operation Describe \
  --topic sales. --resource-pattern-type prefixed

This covers sales.orders, sales.refunds and future topics with that prefix.

#Error-to-ACL map

Warning

Don't fix authorization errors by adding the application to super.users or setting allow.everyone.if.no.acl.found=true. Both switch off security for that principal or for every topic without ACLs.

#Removing ACLs

kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
  --remove --allow-principal User:old-app --operation Read --topic orders --force

#Prevent it next time

• Manage ACLs as code (Terraform Kafka provider, Strimzi KafkaUser, or a reviewed script), not ad-hoc commands.
• Use prefixed ACLs per team or domain with a topic naming convention.
• Set ssl.principal.mapping.rules so certificate principals are short and predictable.
• Ship kafka-authorizer.log to your log platform and alert on new denial patterns after deployments.