Azure Service Bus MessageLockLost: stop long handlers from causing retries and duplicates

#Symptom

Consumers log MessageLockLost ("The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue"). Orders get processed twice, the delivery count climbs, and eventually messages appear in the dead-letter queue with reason MaxDeliveryCountExceeded. The handler itself reported success.

#What's happening

In peek-lock mode, a receiver gets a message with a timed lock. If the lock expires before the receiver completes the message, Service Bus makes it available again and raises the delivery count. Your handler may still finish its work, but its complete call now fails. Another consumer then processes the same message again.

#Step 1: gather the settings

az servicebus queue show -g rg-msg --namespace-name sb-ops -n orders \
  --query "{lock:lockDuration, maxDelivery:maxDeliveryCount, active:countDetails.activeMessageCount, dlq:countDetails.deadLetterMessageCount}"

# For a topic subscription
az servicebus topic subscription show -g rg-msg --namespace-name sb-ops \
  --topic-name events -n billing \
  --query "{lock:lockDuration, maxDelivery:maxDeliveryCount}"

Then compare with how long the handler really takes. Log the elapsed time per message and look at the P99, not the average.

#Step 2: find which cause applies

Prefetch is the one people miss. Prefetched messages are locked when they reach the client's buffer, not when your handler starts. With a large prefetch, slow handlers and a 30-second lock, messages can expire before they're ever handled.

#Step 3: configure the consumer

Raise the lock. The maximum is 5 minutes.

az servicebus queue update -g rg-msg --namespace-name sb-ops -n orders --lock-duration PT5M

For work that takes longer, renew the lock from the client.

.NET:

var processor = client.CreateProcessor("orders", new ServiceBusProcessorOptions
{
    AutoCompleteMessages = false,
    MaxConcurrentCalls = 8,
    PrefetchCount = 0,
    MaxAutoLockRenewalDuration = TimeSpan.FromMinutes(30)
});

processor.ProcessMessageAsync += async args =>
{
    await HandleAsync(args.Message, args.CancellationToken);
    await args.CompleteMessageAsync(args.Message);
};

processor.ProcessErrorAsync += args =>
{
    logger.LogError(args.Exception, "Source {Source}", args.ErrorSource);
    return Task.CompletedTask;
};

Python:

from azure.servicebus import ServiceBusClient, AutoLockRenewer

renewer = AutoLockRenewer(max_lock_renewal_duration=1800)

with ServiceBusClient.from_connection_string(conn) as client:
    with client.get_queue_receiver("orders", prefetch_count=0) as receiver:
        for msg in receiver.receive_messages(max_message_count=1, max_wait_time=30):
            renewer.register(receiver, msg)
            handle(msg)
            receiver.complete_message(msg)

renewer.close()

Warning

Auto-renew is a client-side loop. It depends on the process staying healthy, so it can't save you from a blocked thread pool or a crashed pod. It also doesn't help if the handler outlives the configured renewal limit.

Azure Functions manages renewal through maxAutoLockRenewalDuration in host.json. Also check that the function's own timeout isn't shorter than the longest job.

#Step 4: make processing idempotent

Service Bus delivers at least once, so duplicates can happen even after these fixes. Handle them in the consumer:

-- Record the message ID in the same transaction as the business change
BEGIN TRANSACTION;

INSERT INTO processed_messages (message_id, processed_at)
VALUES (@MessageId, SYSUTCDATETIME());  -- fails on duplicate with a unique key

UPDATE orders SET status = 'PAID' WHERE order_id = @OrderId;

COMMIT;

On a unique key violation, treat the message as already done and complete it. The queue's duplicate detection (requiresDuplicateDetection) helps with duplicate sends within its time window, but it doesn't stop redelivery after a lost lock.

#Step 5: redesign for very long work

If processing takes many minutes, don't hold a lock for all of it:

1. The handler validates the message and writes a job record to a database.
2. It completes the message right away.
3. A worker, such as a Container Apps job or Azure Batch, runs the job and updates its status.

This also makes retries and progress visible.

#Handle what's already in the dead-letter queue

Read the reason before you resubmit anything:

from azure.servicebus import ServiceBusSubQueue

with client.get_queue_receiver("orders", sub_queue=ServiceBusSubQueue.DEAD_LETTER) as dlq:
    for msg in dlq.peek_messages(max_message_count=20):
        print(msg.message_id, msg.dead_letter_reason, msg.delivery_count)

Caution

Resubmitting before the handler is idempotent can repeat side effects such as payments or emails. Fix the consumer first, then replay a small batch and check the results. See dead-letter queue troubleshooting.

#Prevent it next time

• Set an alert on the DeadletteredMessages metric above zero.
• Log the delivery count with every message; values above 1 are an early warning.
• Keep the lock duration above your P99 handling time, and use renewal only for the tail.
• Test a rolling restart during load, and confirm that nothing is processed twice.
• Related: namespace throttling and choosing a messaging service.