Sev1: Service Bus sends fail with QuotaExceeded or ServerBusy. Queue full, throttling and backlog recovery

#Symptoms

• Producers log QuotaExceededException (The maximum entity size has been reached or exceeded) or ServerBusyException / ServiceBusException with reason ServiceBusy or QuotaExceeded.
• Consumers fall behind; active message count climbs for hours.
• Downstream systems (orders, notifications) stop receiving work.

#Step 1: which limit? (5 minutes)

NS=$(az servicebus namespace show -g rg-msg -n sb-prod --query id -o tsv)
az servicebus namespace show -g rg-msg -n sb-prod --query "{sku:sku.name, capacity:sku.capacity}" -o table

az servicebus queue show -g rg-msg --namespace-name sb-prod -n orders \
  --query "{maxMB:maxSizeInMegabytes, sizeBytes:sizeInBytes, active:countDetails.activeMessageCount, dlq:countDetails.deadLetterMessageCount, scheduled:countDetails.scheduledMessageCount}" -o table

az monitor metrics list --resource $NS --metric ThrottledRequests ServerErrors UserErrors \
  --interval PT1M --aggregation Total --offset 1h -o table
az monitor metrics list --resource $NS --metric NamespaceCpuUsage NamespaceMemoryUsage \
  --interval PT1M --aggregation Maximum --offset 1h -o table    # Premium only

#Step 2: restore sending

Entity full: the fastest relief is consuming faster or removing what should not be there.

• Scale consumers out (more instances, higher MaxConcurrentCalls / prefetch where processing is idempotent).
• Check the dead-letter queue. If it holds most of the size, move or archive it (step 3).
• Raise the entity max size where the tier allows (Premium supports larger entities; Standard has a fixed upper limit):

az servicebus queue update -g rg-msg --namespace-name sb-prod -n orders --max-size 81920

Throttled Premium namespace: add messaging units. It is online.

az servicebus namespace update -g rg-msg -n sb-prod --capacity 4

Standard throttling: reduce request rate (batch sends, receive in batches, longer polls), or plan a move to Premium. Standard-to-Premium migration exists in the portal for namespaces; test it first since it changes the namespace behind the same name.

Tip

Producers should treat ServerBusy as retryable with backoff. If they retry immediately in a tight loop, they make the throttling worse.

#Step 3: deal with the dead-letter queue

Peek first, never receive-and-delete blindly:

from azure.identity import DefaultAzureCredential
from azure.servicebus import ServiceBusClient, ServiceBusSubQueue

with ServiceBusClient("sb-prod.servicebus.windows.net", DefaultAzureCredential()) as c:
    with c.get_queue_receiver("orders", sub_queue=ServiceBusSubQueue.DEAD_LETTER) as r:
        for m in r.peek_messages(max_message_count=20):
            print(m.sequence_number, m.dead_letter_reason, m.dead_letter_error_description)

Group by reason. Typical reasons are MaxDeliveryCountExceeded (consumer crashed on it) and TTLExpiredException. Fix the consumer bug, then resubmit good messages to the main queue in batches and archive the rest (blob storage) before completing them.

Caution

Resubmitting thousands of dead-lettered orders can double-process them if the consumer is not idempotent. Confirm idempotency or deduplicate by message ID before replaying.

#Step 4: drain the backlog safely

• Increase consumers gradually and watch the downstream database; a backlog drain can turn a messaging incident into a database incident.
• Use sessions or partition-aware processing if order matters.
• Track progress:

AzureMetrics
| where ResourceProvider == "MICROSOFT.SERVICEBUS" and MetricName == "ActiveMessages"
| summarize max(Maximum) by bin(TimeGenerated, 5m)

#Step 5: confirm

• Producers send without QuotaExceeded or ServerBusy for 30 minutes.
• Active message count trending down; dead-letter count stable.
• Downstream systems receiving work.

#Prevent it next time

• Alerts: active messages above N for 15 minutes, dead-letter count above 0 growing, ThrottledRequests above 0, Premium CPU above 70%.
• Set message TTL and max delivery count deliberately per queue.
• Auto-forward dead letters to a dedicated queue with a consumer that alerts and archives.
• Use Premium for production workloads that need predictable throughput, and review messaging units before seasonal peaks.