Blob object replication lagging or failing: a runbook for pending bytes, Failed status and blocked destinations

Object replication copies block blobs asynchronously from a source container to a destination container, often in another region. It fails quietly. The source write succeeds and the application sees no error. You only find out when a DR test or a downstream reader in the other region finds old or missing data.

This runbook goes in order: measure the lag, find the failed blobs, fix the cause, then re-drive replication.

#How it works (the parts that matter when it breaks)

• The source account needs blob versioning and change feed. The destination needs versioning. Replication is driven from the change feed.
• The policy lives on both accounts with the same policy ID. Rules map one source container to one destination container, optionally with prefix filters and a minimum creation time.
• Only block blobs are replicated. Snapshots and version IDs are not; the destination creates its own versions.
• Once a container is a destination, writes to it fail with 409 Conflict. Reads and deletes are allowed.
• Blobs that existed before the rule are skipped unless you set a copy scope (all objects or from a date).
• The source blob reports its replication status per rule once replication has completed or failed. Until then the status is empty.

#Step 1: measure the lag

The source account exposes two replication metrics, shown in the portal as Operations pending for replication and Bytes pending for replication. Both are split into time buckets (0-5 min, 5-10, 10-15, 15-30, 30 min-2 h, 2-8 h, 8-24 h, >24 h). They are off by default. Enable them per policy in the portal (Object replication blade, Metrics column) or through the REST API (2021-08-01 or later). CLI and PowerShell support for this setting has varied between versions, so check az storage account or-policy update --help for a metrics flag before scripting it.

Get the exact metric IDs on your account:

SRC_ID=$(az storage account show -g rg-data -n stordataweu --query id -o tsv)
az monitor metrics list-definitions --resource "$SRC_ID/blobServices/default" \
  --query "[?contains(name.localizedValue, 'replication')].{id:name.value, name:name.localizedValue}" -o table

If nothing comes back, run the same query against $SRC_ID (the account level). The namespace has differed between API versions.

How to read the buckets:

Alert on the 2-8 h and later buckets being non-zero for longer than your RPO, not on the total.

#Step 2: confirm the policy is intact on both sides

A policy edited or deleted on one side only is a common cause of mass failures.

SRC=stordataweu; DST=stordatasdc; RG=rg-data

az storage account or-policy list -g $RG --account-name $SRC \
  --query "[].{id:policyId, src:sourceAccount, dst:destinationAccount, rules:length(rules)}" -o table
az storage account or-policy list -g $RG --account-name $DST \
  --query "[].{id:policyId, src:sourceAccount, dst:destinationAccount, rules:length(rules)}" -o table

# Compare rule-level detail
az storage account or-policy show -g $RG --account-name $DST --policy-id <policy-id> \
  --query "rules[].{rule:ruleId, src:sourceContainer, dst:destinationContainer, filters:filters}"

# Prerequisites still on?
az storage account blob-service-properties show -g $RG --account-name $SRC \
  --query "{versioning:isVersioningEnabled, changeFeed:changeFeed.enabled}"
az storage account blob-service-properties show -g $RG --account-name $DST \
  --query "{versioning:isVersioningEnabled}"

The policy ID, rules and container pairs must match. The destination account and container must exist. A recreated container with the same name is not automatically back in the policy.

For cross-tenant pairs, check allowCrossTenantReplication. Accounts created after December 2023 default to false. With it set to false, policies must reference full resource IDs rather than account names.

#Step 3: find blobs with a Failed status

For one blob:

az storage blob show --account-name $SRC --container-name invoices --name 2026/11/12/batch-0042.json \
  --auth-mode login --query "objectReplicationSourceProperties[].rules[].{rule:ruleId, status:status}"

For a prefix, read each blob's properties. This makes one HEAD request per blob, so scope it to the prefix and time window you care about:

# python -m pip install azure-identity azure-storage-blob
from azure.identity import DefaultAzureCredential
from azure.storage.blob import ContainerClient

cc = ContainerClient("https://stordataweu.blob.core.windows.net", "invoices",
                     credential=DefaultAzureCredential())
for b in cc.list_blobs(name_starts_with="2026/11/"):
    props = cc.get_blob_client(b.name).get_blob_properties()
    for policy in props.object_replication_source_properties or []:
        for rule in policy.rules:
            if (rule.status or "").lower() != "complete":
                print(b.name, b.blob_tier, policy.policy_id, rule.rule_id, rule.status)

Print the tier with the status. It tells you at once whether archive tier is the cause.

#Step 4: fix the cause

To re-create the policy on the source from the destination's definition:

az storage account or-policy show -g $RG --account-name $DST --policy-id <policy-id> |
  az storage account or-policy create -g $RG --account-name $SRC --policy "@-"

Warning

Deleting and re-creating a policy gives it a new ID and, depending on the copy scope, starts initial replication again. On a large container that is days of extra transactions and egress. Prefer adding or fixing rules on the existing policy.

#Step 5: re-drive failed blobs

Replication is triggered by a new write recorded in the change feed. After fixing the cause, write each failed blob again from your system of record, or copy it server-side from its own version URL (<blob-url>?versionid=<id>) onto the same name, which is the documented way to promote a version. Test on one blob first and confirm that its status changes to Complete. Measure the cost first: every rewrite creates a new version on both accounts and counts as a transaction and egress.

Keep the list from step 3 and re-run the check afterwards. Done means the list is empty, not that the metric has gone down.

#Faster replication and its limits

Priority replication can be enabled on one policy per source account. It turns on the metrics automatically and offers an SLA for same-continent pairs: 99% of objects within 15 minutes for supported workloads. It costs extra per GB ingested, and billing continues for 30 days after you disable it. The SLA excludes objects over 5 GB, objects modified more than ten times a second, very high request or throughput rates, and the initial copy. Check the current terms before you rely on it for an RPO.

az storage account or-policy update -g $RG --account-name $SRC --policy-id <policy-id> \
  --priority-replication true

#Things object replication will not do for DR

• Customer-managed account failover is not supported on accounts in a replication policy. Pick one mechanism per account.
• The destination is write-blocked. To fail over writes, delete the rule for that container (or the policy) and point writers at the destination. Agree in advance how and when you set up replication back to the original region.
• Tier changes on the source are not mirrored. Run lifecycle management separately on each side.

#Prevent it next time

• Alert on the late lag buckets (2-8 h and later) against your RPO, per policy.
• Never let lifecycle rules move replicated prefixes to Archive on either account.
• Block CPK uploads to replicated containers in code review, and enforce allowCrossTenantReplication=false with Azure Policy unless you need cross-tenant pairs.
• Manage policies as code and deploy them to both accounts together, so one side cannot drift.
• Include a monthly spot check: write a canary blob, then confirm it reaches Complete and appears in the destination.