Azure Event Grid events not delivered: subscriptions, filters, validation, retries and dead-lettering

#Symptom

A blob lands in storage, or your app publishes to a custom topic, but the Function, webhook or queue on the other side never sees the event. Or it sees some events, late, or twice.

Event Grid gives you enough metrics to find out which stage is failing in a few minutes. Work through them in order: published → matched → delivered → failed/dropped/dead-lettered.

#Step 1: Find the topic and its subscriptions

Storage accounts, resource groups and subscriptions publish through system topics. Your own apps publish to custom topics.

# System topics in a resource group
az eventgrid system-topic list -g rg-data -o table

# Subscriptions on a system topic
az eventgrid system-topic event-subscription list \
  -g rg-data --system-topic-name st-data-events -o table

# Subscriptions on any resource (custom topic or the source resource itself)
az eventgrid event-subscription list \
  --source-resource-id "/subscriptions/<sub>/resourceGroups/rg-data/providers/Microsoft.Storage/storageAccounts/stdata" -o table

If nothing is listed, the subscription was never created or was deleted with its source. Also check the resource provider:

az provider show --namespace Microsoft.EventGrid --query registrationState -o tsv

It must be Registered in the subscription that owns the source. Register it with az provider register --namespace Microsoft.EventGrid.

#Step 2: Read the delivery metrics

TOPIC_ID=$(az eventgrid system-topic show -g rg-data -n st-data-events --query id -o tsv)

az monitor metrics list --resource $TOPIC_ID \
  --metric PublishSuccessCount MatchedEventCount DeliverySuccessCount \
           DeliveryAttemptFailCount DroppedEventCount DeadLetteredCount \
  --interval PT5M --aggregation Total --offset 2h -o table

Add --filter "EventSubscriptionName eq '*'" to split the delivery metrics per subscription when a topic has several. Metric names can differ slightly between topic types; run az monitor metrics list-definitions --resource $TOPIC_ID -o table to see what yours exposes.

#Step 3: Nothing is being published

• Storage events: Blob events need a general-purpose v2 or Blob Storage account. On accounts with a hierarchical namespace (Data Lake Gen2), BlobCreated fires when the file is flushed and closed, so tools that write in several steps raise the event at the end, not at the start.
• Custom topics: Check the publisher's response. A 401 means a wrong key or missing EventGrid Data Sender role; a 403 often means the topic's public network access is disabled or the client IP isn't allowed. Look at PublishFailCount on the topic.
• Private topics: If the topic only allows private endpoints, the publisher must resolve the topic hostname to the private IP.

#Step 4: Events published but not matched

Show the subscription's filter:

az eventgrid system-topic event-subscription show -g rg-data \
  --system-topic-name st-data-events -n sub-ingest \
  --query "{types:filter.includedEventTypes, begins:filter.subjectBeginsWith, ends:filter.subjectEndsWith, advanced:filter.advancedFilters}" -o json

Common mistakes:

• subjectBeginsWith uses the full subject path. For blobs that's /blobServices/default/containers/<container>/blobs/<prefix>, not just the container name.
• Subject matching is case-insensitive by default. If someone set isSubjectCaseSensitive to true, .CSV no longer matches .csv.
• includedEventTypes lists Microsoft.Storage.BlobCreated, but the app expects BlobRenamed or BlobDeleted too.
• Advanced filters on data.* keys that don't exist in that event schema never match.

Fix the filter in place:

az eventgrid system-topic event-subscription update -g rg-data \
  --system-topic-name st-data-events -n sub-ingest \
  --subject-begins-with /blobServices/default/containers/landing/blobs/ \
  --subject-ends-with .csv

#Step 5: Delivery attempts failing

Turn on diagnostic logs for the topic (category DeliveryFailures) and send them to Log Analytics. Depending on the destination mode, the rows land in a resource-specific table such as AegDeliveryFailureLogs or in AzureDiagnostics. Start broad, then narrow down:

AegDeliveryFailureLogs
| where TimeGenerated > ago(2h)
| take 50

The error text and HTTP status tell you which row of this table applies:

For Azure Functions, prefer the azurefunction endpoint type over the webhook URL. It uses the function's resource ID, survives key rotation and doesn't need the handshake:

az eventgrid system-topic event-subscription update -g rg-data \
  --system-topic-name st-data-events -n sub-ingest \
  --endpoint-type azurefunction \
  --endpoint "/subscriptions/<sub>/resourceGroups/rg-fn/providers/Microsoft.Web/sites/fn-ingest/functions/OnBlobCreated"

#Step 6: Dropped and dead-lettered events

Event Grid retries with exponential backoff until either the maximum delivery attempts (default 30) or the event time-to-live (default 1,440 minutes) is reached, whichever comes first. Dead-lettering is off by default, so without it an event that runs out of retries is simply dropped and counted in DroppedEventCount.

Caution

If DroppedEventCount is above zero and no dead-letter destination is set, those events are gone. Publishers don't resend them. You must rebuild the missed work from the source, for example by listing blobs created in the outage window.

Turn on dead-lettering to a storage container:

DLQ="/subscriptions/<sub>/resourceGroups/rg-data/providers/Microsoft.Storage/storageAccounts/stdlq/blobServices/default/containers/eventgrid-dlq"

az eventgrid system-topic event-subscription update -g rg-data \
  --system-topic-name st-data-events -n sub-ingest \
  --deadletter-endpoint $DLQ

Set --max-delivery-attempts and --event-ttl (minutes) on create if your handler can't use events after a certain age. Events reach the dead-letter container a few minutes after the last attempt, as JSON blobs with the delivery error attached. Once the handler is fixed, replay them from the blob to the topic or straight to the handler.

#Prevent it next time

• Alert on DroppedEventCount > 0, DeadLetteredCount > 0 and a rising DeliveryAttemptFailCount for every production subscription.
• Always configure a dead-letter container. It's the only record of events that failed.
• Keep handlers fast: acknowledge, enqueue, process. Anything that may take more than a few seconds belongs behind a queue.
• Make handlers idempotent. Event Grid delivers at least once, so duplicates are normal after retries.
• Manage subscriptions in Bicep or Terraform. A subscription deleted with its source resource or recreated by hand with the wrong filter is a common cause of silent gaps.