Sev1: Event Hubs consumers stuck — ReceiverDisconnected, epoch conflicts and lost checkpoints

#Symptoms

• Logs full of ReceiverDisconnectedException / "New receiver with higher epoch ... created hence current receiver ... is getting disconnected".
• Some partitions not processed; lag grows on a few partitions only.
• After a redeploy, consumers start from the beginning (duplicates) or the end (data missed).

#1. Who else is reading this consumer group?

Epoch errors mean two different processors claim the same partition in the same consumer group — usually two apps sharing $Default, or old and new versions of a deployment running at once.

Fix: one consumer group per application.

az eventhubs eventhub consumer-group list -g rg-stream --namespace-name evhns-prod --eventhub-name telemetry -o table
az eventhubs eventhub consumer-group create -g rg-stream --namespace-name evhns-prod --eventhub-name telemetry -n cg-orders-loader

#2. Inspect the checkpoint store

EventProcessorClient stores ownership and checkpoints as blobs: <namespace>/<eventhub>/<consumergroup>/ownership/<partition> and /checkpoint/<partition>.

az storage blob list --account-name stcheckpoints -c eh-checkpoints --auth-mode login \
  --prefix "evhns-prod.servicebus.windows.net/telemetry/cg-orders-loader/" \
  --query "[].{name:name, modified:properties.lastModified}" -o table

• Ownership blobs updated by many instances in quick succession = ownership "thrashing" (instances fighting; check load-balancing settings and instance count vs. partitions).
• Missing checkpoint blobs = the processor starts from its default position (configurable start position, often latest).

#3. Starting position after lost checkpoints

Set the start position explicitly when checkpoints are missing, for example to a time just before the incident (EventPosition.FromEnqueuedTime(...) in .NET, starting_position in Python). Make the sink idempotent before replaying.

Warning

Event Hubs retention is limited (check your tier). Data older than retention is gone; replay only works within it. Capture to storage gives you a longer replay source.

#4. Stalled partitions

• One instance processing too slowly: check per-partition lag; a poison event may throw on every retry. Log and dead-letter it instead of failing the batch forever.
• Functions trigger: check maxEventBatchSize, function timeout, and that the function app isn't throttled.
• More consumer instances than partitions don't help — extra instances sit idle.

#5. Check the client versions

Old SDKs (pre-EventProcessorClient EventProcessorHost) use different checkpoint formats. Mixing old and new versions on the same consumer group and storage leads to conflicts. Migrate all instances together, with a new consumer group if needed.

#Prevent it next time

• One consumer group per app; never use $Default in production.
• Graceful shutdown in deployments so old instances release ownership.
• Alerts on per-partition lag (consumer lag metrics or your own) and on error rate.
• Event Hubs Capture for important streams to allow replay beyond retention.