Kafka client quotas: stop one noisy producer or consumer from starving the cluster, and diagnose throttling

#Goal

On a shared Kafka cluster, one badly behaved client can take the cluster down for everyone: a backfill job that produces at full network speed, a consumer replaying a topic from the earliest offset, or a script opening thousands of connections. Client quotas cap each client so the broker slows it down instead of letting it saturate disks, network and request threads.

This post covers setting quotas, checking what is in force, and the other side of the problem: a client that suddenly got slower because a quota is throttling it.

#How quotas work

• Quotas are per broker. A producer_byte_rate of 10 MB/s means 10 MB/s to each broker, so a client spread over 6 brokers can reach about 60 MB/s in total.
• The broker does not reject throttled requests. It works out how long the client must wait to come back under the limit, returns the response with a throttle_time_ms, and holds off reading more from that connection. The client sees higher latency, not errors.
• Quotas apply to an entity: a user (authenticated principal), a client ID, or a user plus client ID pair. Without authentication every client is user ANONYMOUS, so only client ID quotas are useful.

#Which quota wins

When several match, the most specific one applies, in this order:

1. user + client ID
2. user + default client ID
3. user
4. default user + client ID
5. default user + default client ID
6. default user
7. client ID
8. default client ID

A quota on the exact user and client ID always overrides a broad default, which is how you make an exception for one heavy job.

#Set quotas

The commands below use kafka-configs.sh (kafka-configs on Confluent Platform). Add --command-config admin.properties when the cluster requires authentication.

#Per user

kafka-configs.sh --bootstrap-server broker1:9092 --alter \
  --entity-type users --entity-name svc-orders-etl \
  --add-config 'producer_byte_rate=10485760,consumer_byte_rate=20971520,request_percentage=50'

#Per client ID (no authentication, or to split one user's apps)

kafka-configs.sh --bootstrap-server broker1:9092 --alter \
  --entity-type clients --entity-name nightly-backfill \
  --add-config 'producer_byte_rate=5242880'

#User plus client ID

kafka-configs.sh --bootstrap-server broker1:9092 --alter \
  --entity-type users --entity-name svc-orders-etl \
  --entity-type clients --entity-name orders-replay \
  --add-config 'consumer_byte_rate=5242880'

#A default for everyone

kafka-configs.sh --bootstrap-server broker1:9092 --alter \
  --entity-type users --entity-default \
  --add-config 'producer_byte_rate=20971520,consumer_byte_rate=41943040'

Warning

A default user quota also applies to internal and platform principals: Kafka Connect workers, MirrorMaker 2, Schema Registry, Kafka Streams apps and your monitoring. Set explicit, higher quotas for those principals before you add the default, or replication pipelines and Connect tasks will start lagging the moment it lands.

#Limit topic creation and connection storms

# A CI job that creates and deletes topics
kafka-configs.sh --bootstrap-server broker1:9092 --alter \
  --entity-type users --entity-name svc-ci \
  --add-config 'controller_mutation_rate=10'

# Cap new connections from any single IP
kafka-configs.sh --bootstrap-server broker1:9092 --alter \
  --entity-type ips --entity-default \
  --add-config 'connection_creation_rate=20'

When a client exceeds controller_mutation_rate, CreateTopics, CreatePartitions and DeleteTopics requests fail with THROTTLING_QUOTA_EXCEEDED (newer admin clients retry automatically). This is the one quota that returns an error rather than just delaying.

#Check what is in force

# All user quotas, including defaults
kafka-configs.sh --bootstrap-server broker1:9092 --describe --entity-type users

# All client ID quotas
kafka-configs.sh --bootstrap-server broker1:9092 --describe --entity-type clients

# One user
kafka-configs.sh --bootstrap-server broker1:9092 --describe \
  --entity-type users --entity-name svc-orders-etl

# IP quotas
kafka-configs.sh --bootstrap-server broker1:9092 --describe --entity-type ips

Remove a quota with --delete-config:

kafka-configs.sh --bootstrap-server broker1:9092 --alter \
  --entity-type clients --entity-name nightly-backfill \
  --delete-config 'producer_byte_rate'

Changes take effect on all brokers within seconds, with no restart. Quotas are stored in cluster metadata (KRaft) or ZooKeeper, so they survive restarts.

#Diagnose: is the client throttled, or is the broker slow?

A team reports their producer or consumer "got slow". Check throttle metrics before you look at broker hardware.

#Client-side metrics

Non-zero values mean a quota is active. Zero throttle time with high request-latency-avg points at the broker, the network or the client itself.

#Broker-side metrics

The broker publishes per-entity quota metrics under kafka.server:

• kafka.server:type=Produce,user=<user>,client-id=<client> with byte-rate and throttle-time
• kafka.server:type=Fetch,user=<user>,client-id=<client> with byte-rate and throttle-time
• kafka.server:type=Request,user=<user>,client-id=<client> with request-time and throttle-time

The tag set depends on which quota level matched, so a client-ID-only quota shows user="". Exact MBean names can vary by version; browse them with JConsole or your JMX exporter's output before writing alerts.

#Find who is near the limit

Compare each principal's byte-rate against its quota. A client sitting at the quota for hours is either correctly capped or under-provisioned; ask the owner which.

Tip

Keep client.id set and meaningful in every application (orders-api-producer, not the default producer-1). Without it, client ID quotas and per-client metrics are useless, and you can't tell which app is hitting the limit.

#Rollout plan for a cluster with no quotas

1. Make sure clients authenticate as distinct principals and set client.id.
2. Collect two weeks of per-principal byte-rate from the broker metrics.
3. Set explicit quotas for platform principals (Connect, MirrorMaker 2, Streams apps) above their observed peak.
4. Set a default user quota well above normal application peaks, so it only catches runaway clients.
5. Alert on client throttle time above zero for more than a few minutes, so you hear about throttling before the application team does.
6. Tighten defaults gradually, and document how teams request an exception.

Caution

Don't apply a tight default quota during an incident to stop a noisy client. It throttles every client, including the ones keeping production running. Put the quota on the offending user or client ID only.

#Prevent it next time

• Treat quotas as config in Git and apply them with a script, so they are reviewed and reproducible after a cluster rebuild.
• Give backfill and replay jobs their own client ID with their own quota, set before the job starts.
• Add produce and fetch throttle time to the standard client dashboard.
• Recheck quotas when you add brokers: per-broker quotas mean the effective cluster-wide limit grows with the broker count.
• On managed services, check the current limits and which quota types they expose before planning around these settings.