Event Hubs Capture not writing files: storage access, firewalls, empty files and backlog

#The symptom

Event Hubs Capture is enabled, producers are sending, and one of these is true:

• The destination container has no new blobs.
• Blobs appear, but most are tiny Avro files with no events in them.
• Files arrive, but the newest one is hours behind the events being sent.
• Downstream jobs (Spark, ADF, Synapse) fail to find the path they expect.

Capture writes Avro files per partition, on whichever comes first: the time window or the size window. If either the config, the storage permission or the storage network blocks it, Capture stops writing but the event hub keeps accepting events. Data is not lost straight away. Microsoft documents that if storage is temporarily unavailable, Capture keeps the data for the event hub's retention period and backfills once the storage is reachable again. That retention period is your deadline.

Warning

Once events age past the event hub's retention period without being captured, they are gone from Capture's point of view. Check the retention setting first and fix the cause before it runs out.

#Quick checks

#Step 1: Read the actual Capture config

RG=rg-streaming
NS=evhns-prod
EH=orders

az eventhubs eventhub show -g $RG --namespace-name $NS -n $EH \
  --query "{retention:retentionDescription, messageRetentionInDays:messageRetentionInDays, partitions:partitionCount, capture:captureDescription}" -o json

az eventhubs namespace show -g $RG -n $NS --query "{sku:sku.name, identity:identity}" -o json

Look at:

• capture.enabled: must be true.
• capture.destination.storageAccountResourceId and blobContainer: the account and container you are actually looking at. A copy-pasted dev account ID is a common find.
• capture.intervalInSeconds and sizeLimitInBytes: the time and size windows.
• capture.skipEmptyArchives: whether idle partitions write empty files.
• capture.destination.archiveNameFormat: the blob path pattern.
• sku: Basic does not support Capture. Standard, Premium and Dedicated do.

Field names in the JSON vary a little between API and CLI versions; check your own output. Retention is shown as messageRetentionInDays on older versions and inside retentionDescription on newer ones.

#Step 2: See what has been written

List the newest blobs for one partition. The default style of path is {Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}, with zero-padded date parts.

SA=stlakeprod
CONTAINER=capture

az storage blob list --account-name $SA --container-name $CONTAINER \
  --prefix "$NS/$EH/0/$(date -u +%Y/%m/%d)/" --auth-mode login \
  --query "sort_by([], &properties.lastModified)[-10:].{name:name, bytes:properties.contentLength, modified:properties.lastModified}" \
  -o table

If the last file is from the moment someone changed storage networking or RBAC, you have your cause. If files are recent but tiny, go to the empty-files section.

Tip

--auth-mode login uses your Entra ID identity, so you need a data-plane role such as Storage Blob Data Reader on the account. A management-plane Owner role alone is not enough to list blobs this way.

#Step 3: Check the backlog and errors

Capture exposes namespace metrics for what it has written and what it still has to write. Names vary by version; list them first:

NS_ID=$(az eventhubs namespace show -g $RG -n $NS --query id -o tsv)

az monitor metrics list-definitions --resource $NS_ID \
  --query "[?contains(name.value, 'Captur')].name.value" -o tsv

Then pull the backlog for the event hub, typically CaptureBacklog, alongside CapturedMessages and IncomingMessages:

az monitor metrics list --resource $NS_ID \
  --metric CaptureBacklog CapturedMessages IncomingMessages \
  --interval PT5M --aggregation Total \
  --filter "EntityName eq '$EH'" -o table

A backlog that grows while CapturedMessages is zero means Capture is failing to write. A backlog that is shrinking means it is catching up, so leave it alone.

For the reason, send the namespace's Capture logs (the ArchiveLogs category) to Log Analytics if you haven't already:

az monitor diagnostic-settings categories list --resource $NS_ID -o table

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.EVENTHUB"
| where Category == "ArchiveLogs"
| where TimeGenerated > ago(6h)
| project TimeGenerated, Resource, EventHub = column_ifexists("eventHub_s", ""),
          Message = column_ifexists("message_s", ""), Failures = column_ifexists("failures_d", real(null))
| order by TimeGenerated desc

Column names depend on whether you use Azure diagnostics mode or resource-specific tables; run AzureDiagnostics | where Category == "ArchiveLogs" | take 5 first and adjust. Authorization and network errors show up here.

#Step 4: Fix storage access

#Managed identity

If Capture uses a managed identity, that identity needs a data role on the destination. Microsoft's Capture guidance uses Storage Blob Data Contributor at the account or container scope.

PRINCIPAL=$(az eventhubs namespace show -g $RG -n $NS --query identity.principalId -o tsv)
SA_ID=$(az storage account show -g $RG -n $SA --query id -o tsv)

az role assignment list --assignee $PRINCIPAL --scope $SA_ID -o table

az role assignment create --assignee $PRINCIPAL \
  --role "Storage Blob Data Contributor" --scope $SA_ID

Role assignments can take several minutes to apply. For a user-assigned identity, use that identity's principal ID instead of the namespace's.

#Without managed identity

The person or service principal who enables or updates Capture needs write permission on containers and blobs in the storage account, and the storage account must be in the same subscription as the event hub. Moving the storage account to another subscription breaks this setup; switch to a managed identity.

#Storage firewall

If the storage account has public network access restricted, Capture needs the trusted Microsoft services exception:

az storage account show -g $RG -n $SA --query "{publicAccess:publicNetworkAccess, rules:networkRuleSet}" -o json

az storage account update -g $RG -n $SA --bypass AzureServices

Caution

If --bypass already includes Logging or Metrics, pass them together (--bypass AzureServices Logging Metrics). The flag replaces the list, so setting only AzureServices removes the others.

Also check that the destination is a standard (non-premium) account that supports block blobs. Premium storage accounts are not supported as a Capture destination.

#Step 5: Fix empty files and window sizes

By design, Capture writes empty files when a partition has no data in a window, so downstream systems can see it is still working. With 32 partitions and a 60-second window, that is a lot of empty blobs. Turn it off if your readers don't depend on them:

az eventhubs eventhub update -g $RG --namespace-name $NS -n $EH \
  --skip-empty-archives true

Tune the windows to balance file size against latency. The CLI accepts a time window of 60 to 900 seconds and a size window of 10,485,760 to 524,288,000 bytes:

az eventhubs eventhub update -g $RG --namespace-name $NS -n $EH \
  --capture-interval 300 --capture-size-limit 314572800

Bigger windows mean fewer, larger files and fewer small-file problems for Spark. They also mean data lands up to one window later.

#Step 6: Check the path the readers expect

If you change --archive-name-format, every placeholder ({Namespace}, {EventHub}, {PartitionId}, {Year}, {Month}, {Day}, {Hour}, {Minute}, {Second}) must still be present, in any order. New files go to the new layout and old files stay where they were, so readers with a hard-coded glob will miss one or the other.

To confirm the files themselves are readable, open one in Spark:

df = (spark.read.format("avro")
      .load("abfss://capture@stlakeprod.dfs.core.windows.net/evhns-prod/orders/0/2026/11/09/*/*/*.avro"))

(df.selectExpr("SequenceNumber", "EnqueuedTimeUtc", "CAST(Body AS STRING) AS body")
   .orderBy("SequenceNumber", ascending=False)
   .show(5, truncate=80))

Body is binary, so cast it or decode it with your producer's format.

#Prevent it next time

• Alert on the Capture backlog metric for each captured event hub, not just on incoming messages.
• Keep event hub retention longer than your realistic time to fix a storage problem, so Capture can backfill.
• Use a managed identity with an explicit role assignment, and record it in your infrastructure code so a storage rebuild doesn't drop it.
• Make storage network changes go through a check that the AzureServices bypass or a private path for Capture is still in place.
• Set skipEmptyArchives and window sizes on purpose and document the path format readers depend on.