Azure Service Bus dead-letter queue filling up - finding why messages fail and resubmitting them safely

#Symptoms

• DeadLetteredMessages count on a queue or subscription keeps climbing.
• Downstream systems are missing orders, events or updates.
• Consumers log exceptions, or they look healthy but process nothing.

Every queue and subscription has a dead-letter sub-queue (<queue>/$DeadLetterQueue). Messages stay there until something reads them. They don't expire on their own.

#1. How big is the problem?

az servicebus queue show -g rg-msg --namespace-name sb-prod -n orders \
  --query "countDetails" -o json

az servicebus topic subscription show -g rg-msg --namespace-name sb-prod \
  --topic-name events -n billing --query "countDetails" -o json

countDetails shows activeMessageCount, deadLetterMessageCount, scheduledMessageCount and transferDeadLetterMessageCount.

In Azure Monitor, chart the DeadletteredMessages metric split by entity name to see when the growth started. That time usually matches a deployment or a downstream outage.

#2. Why were they dead-lettered?

Each dead-lettered message carries DeadLetterReason and DeadLetterErrorDescription properties.

Peek (don't receive) a sample with Python:

from azure.identity import DefaultAzureCredential
from azure.servicebus import ServiceBusClient, ServiceBusSubQueue

ns = "sb-prod.servicebus.windows.net"
with ServiceBusClient(ns, DefaultAzureCredential()) as client:
    with client.get_queue_receiver("orders", sub_queue=ServiceBusSubQueue.DEAD_LETTER) as r:
        for m in r.peek_messages(max_message_count=20):
            print(m.sequence_number, m.dead_letter_reason, m.dead_letter_error_description,
                  m.delivery_count, str(m)[:200])

The identity needs the Azure Service Bus Data Receiver role on the namespace or entity.

Group the sample by reason. One dominant reason means one root cause.

#3. Fix the cause before you resubmit

• MaxDeliveryCountExceeded: check consumer logs at the time of failure. Common causes are poison messages (bad schema), a downstream API returning errors, or a lock timeout because processing takes longer than LockDuration.
• Check the entity's settings:

az servicebus queue show -g rg-msg --namespace-name sb-prod -n orders \
  --query "{maxDelivery:maxDeliveryCount, lock:lockDuration, ttl:defaultMessageTimeToLive, dlqOnExpiry:deadLetteringOnMessageExpiration}"

• If processing is slow, either renew the lock in the consumer (the SDKs have auto lock renewal) or raise lockDuration (maximum 5 minutes).

Warning

Don't raise maxDeliveryCount to hide a poison message. It just makes each bad message block a consumer for longer.

#4. Resubmit safely

Resubmitting means: receive from the DLQ, send a copy to the main queue, then complete the DLQ message. Do it in small batches and only after the consumer is fixed.

from azure.servicebus import ServiceBusMessage

with ServiceBusClient(ns, DefaultAzureCredential()) as client:
    sender = client.get_queue_sender("orders")
    with client.get_queue_receiver("orders", sub_queue=ServiceBusSubQueue.DEAD_LETTER,
                                   max_wait_time=5) as r, sender:
        moved = 0
        for m in r.receive_messages(max_message_count=50):
            if m.dead_letter_reason != "MaxDeliveryCountExceeded":
                continue  # leave other reasons for manual review
            copy = ServiceBusMessage(body=b"".join(m.body),
                                     application_properties=m.application_properties,
                                     correlation_id=m.correlation_id,
                                     message_id=m.message_id,
                                     session_id=m.session_id,
                                     content_type=m.content_type)
            sender.send_messages(copy)
            r.complete_message(m)
            moved += 1
        print("resubmitted", moved)

The sender needs the Data Sender role. Keep the original message_id if duplicate detection is on, but note that duplicate detection may then drop the resubmitted copy if it is still inside the detection window.

Caution

Make sure consumers are idempotent before resubmitting. Some dead-lettered messages may already have been partly processed, for example a payment written before the failure.

Service Bus Explorer in the portal can also resubmit DLQ messages one at a time or in small selections. It's useful for a handful of messages.

#5. Topic subscriptions

The same applies to subscriptions, using get_subscription_receiver(topic, subscription, sub_queue=...). Resubmit to the topic, which delivers the message to every subscription again. If only one subscription failed, send to a dedicated retry subscription or use a filter. Otherwise the other subscribers get duplicates.

#Prevent it next time

• Alert when DeadletteredMessages is above zero for more than 15 minutes, per entity.
• Validate the message schema in the consumer and dead-letter bad messages with a clear custom reason.
• Size lockDuration for the slowest normal processing time, or use automatic lock renewal.
• Document a resubmit procedure and who approves it.

#Checklist

• DLQ counts and start time known
• Reasons grouped from a peeked sample
• Consumer or downstream fixed
• Idempotency confirmed
• Resubmitted in small batches, counts verified
• DLQ alert in place