Home / Data Platform / Data Factory

Data PlatformRunbookIntermediate

Azure Data Factory pipeline failure runbook

Find failed runs across factories with KQL, read activity error codes, fix self-hosted integration runtime problems and rerun from the point of failure.

Deepak Bhardwaj3 min read

Applies toAzure Data Factory, Synapse pipelines, Fabric Data Factory (concepts)

1. Find what failed

With diagnostic settings sending logs to Log Analytics (use resource-specific tables):

ADFActivityRun
| where TimeGenerated > ago(24h) and Status == "Failed"
| project TimeGenerated, PipelineName, ActivityName, ActivityType, ErrorCode, ErrorMessage = tostring(Error), PipelineRunId
| order by TimeGenerated desc

// Which pipelines fail most this week?
ADFPipelineRun
| where TimeGenerated > ago(7d)
| summarize runs = count(), failed = countif(Status == "Failed") by PipelineName
| extend failRate = round(100.0 * failed / runs, 1)
| order by failed desc

From the CLI:

az datafactory pipeline-run query-by-factory -g rg-data --factory-name adf-prod \
  --last-updated-after 2026-10-04T00:00:00Z --last-updated-before 2026-10-05T23:59:59Z \
  --filters operand=Status operator=Equals values=Failed \
  --query "value[].{pipeline:pipelineName,run:runId,msg:message}" -o table

2. Read the error code

Error Usual cause Fix
2200 / UserErrorFailedToConnectToSqlServer Firewall, private endpoint DNS, or wrong credential Test the linked service, check DNS resolution from the IR
UserErrorSourceBlobNotExists Upstream file is late or the path changed Add a Get Metadata/Validation activity, or use event triggers
ErrorCode=SqlOperationFailed ... deadlock / timeout Contention on the sink Lower parallel copies, use a staging table and then MERGE
TypeConversionFailure Schema drift in the source Explicit mapping, or set enableSkipIncompatibleRow with a logging path
Self-hosted integration runtime ... offline / TaskTimeout IR node down or overloaded See step 3
Forbidden/403 on storage The managed identity lacks the RBAC role Grant Storage Blob Data Contributor on the container

3. Self-hosted integration runtime health

On the IR VM:

Get-Service DIAHostService                       # should be Running
& "C:\Program Files\Microsoft Integration Runtime\5.0\Shared\dmgcmd.exe" -Status
Test-NetConnection <sql-server>.database.windows.net -Port 1433
Resolve-DnsName <storage>.blob.core.windows.net   # private endpoint? expect 10.x
  • Check CPU and memory on the node. If concurrent jobs are near the limit, scale out to a second node instead of up.
  • Keep the IR version current. Auto-update can be turned off by mistake.
  • If outbound traffic goes through a proxy, allow *.servicebus.windows.net and *.frontend.clouddatahub.net.

4. Rerun safely

  • Rerun from failed activity (Monitor → pipeline run → Rerun from failed activity) skips the steps that already succeeded.
  • Make sure copy sinks are idempotent: overwrite by partition, or MERGE on keys, so a rerun doesn't create duplicates.
  • For tumbling window triggers, rerun the specific window so watermarks stay correct.

Tip

Make the failure alert useful: route it through an Action Group to a Logic App that posts the pipeline name, activity, error code and a deep link to the run in Teams.

5. Prevent it next time

  • Set a retry (2–3 tries, 60–120 s interval) on copy and lookup activities that talk to external systems.
  • Set an activity timeout. The default is 12 hours, which is far too long for most copies.
  • Track source file arrival SLAs separately from pipeline failures.

Related fixes