Azure Data Factory operations cheat sheet: find, rerun, cancel and pause pipelines from the CLI, PowerShell and KQL

The portal Monitor page is fine for one run. When twenty pipelines failed overnight, or you need to pause every trigger before a maintenance window, the command line is faster and leaves a record. This sheet groups the commands by the job you are doing at the time.

Set these once per shell so the commands below stay short:

az extension add --name datafactory --upgrade
RG=rg-data-prod
DF=adf-data-prod

Install-Module Az.DataFactory -Scope CurrentUser   # once
$rg = "rg-data-prod"; $df = "adf-data-prod"

Tip

The datafactory CLI extension and the Az.DataFactory module both change over time. If a flag below is rejected, run az datafactory <group> <command> --help or Get-Help <cmdlet> -Parameter * to see what your installed version supports.

#Find what failed

#Failed pipeline runs in the last 24 hours (Azure CLI)

query-by-factory needs both --last-updated-after and --last-updated-before in ISO 8601 UTC.

AFTER=$(date -u -d '-24 hours' +%Y-%m-%dT%H:%M:%SZ)
BEFORE=$(date -u +%Y-%m-%dT%H:%M:%SZ)

az datafactory pipeline-run query-by-factory \
  --resource-group $RG --factory-name $DF \
  --last-updated-after $AFTER --last-updated-before $BEFORE \
  --filters operand="Status" operator="Equals" values="Failed" \
  --query "value[].{pipeline:pipelineName, runId:runId, start:runStart, msg:message}" -o table

Valid filter operands for pipeline runs include PipelineName, RunStart, RunEnd and Status. Results are paged; if the output contains a continuationToken, pass it back with --continuation-token to get the next page.

#Same thing in PowerShell

Get-AzDataFactoryV2PipelineRun -ResourceGroupName $rg -DataFactoryName $df `
  -LastUpdatedAfter (Get-Date).AddDays(-1) -LastUpdatedBefore (Get-Date) |
  Where-Object Status -eq 'Failed' |
  Select-Object PipelineName, RunId, RunStart, Message

#Which activity failed and why

az datafactory activity-run query-by-pipeline-run \
  --resource-group $RG --factory-name $DF --run-id <pipeline-run-id> \
  --last-updated-after $AFTER --last-updated-before $BEFORE \
  --query "value[?status=='Failed'].{activity:activityName, type:activityType, error:error.message}" -o table

Get-AzDataFactoryV2ActivityRun -ResourceGroupName $rg -DataFactoryName $df `
  -PipelineRunId <pipeline-run-id> `
  -RunStartedAfter (Get-Date).AddDays(-1) -RunStartedBefore (Get-Date) |
  Where-Object Status -eq 'Failed' |
  Select-Object ActivityName, ActivityType, @{n='Error';e={$_.Error.message}}

The activity error message is usually the only thing you need. For a Copy activity, the output property also holds rows read and written, throughput and the integration runtime that ran it.

#Search across factories with KQL

If the factory sends diagnostic logs to Log Analytics in resource-specific mode, you get the ADFPipelineRun, ADFActivityRun and ADFTriggerRun tables. This is the only practical way to search many factories at once.

# Turn on diagnostics (one-off). Resource-specific tables, not AzureDiagnostics.
az monitor diagnostic-settings create --name adf-to-law \
  --resource $(az datafactory show -g $RG -n $DF --query id -o tsv) \
  --workspace <log-analytics-workspace-resource-id> \
  --export-to-resource-specific true \
  --logs '[{"category":"PipelineRuns","enabled":true},{"category":"ActivityRuns","enabled":true},{"category":"TriggerRuns","enabled":true}]'

Failed activities with their error, last 24 hours:

ADFActivityRun
| where TimeGenerated > ago(24h)
| where Status == "Failed"
| project TimeGenerated, _ResourceId, PipelineName, ActivityName, ActivityType, ErrorCode, ErrorMessage, PipelineRunId
| order by TimeGenerated desc

Top error codes this week, so you fix the cause rather than each run:

ADFActivityRun
| where TimeGenerated > ago(7d) and Status == "Failed"
| summarize Failures = count(), Pipelines = dcount(PipelineName), Sample = any(ErrorMessage) by ErrorCode, ActivityType
| order by Failures desc

Pipelines that have not succeeded since yesterday (catches "it never ran" as well as "it failed"):

ADFPipelineRun
| where TimeGenerated > ago(3d)
| summarize LastSuccess = maxif(End, Status == "Succeeded"), LastRunSeen = max(TimeGenerated) by PipelineName, _ResourceId
| where LastSuccess < ago(1d) or isnull(LastSuccess)

Diagnostic data usually lands a few minutes after the run ends, so do not use these queries to judge a run that finished seconds ago.

#Rerun

When you pass a reference run ID, the new run reuses the parameters of that run, and recovery mode groups the old and new runs under the same group ID in Monitor. create-run returns the new runId; save it.

Warning

Rerunning from failure skips activities that succeeded. If an earlier activity wrote partial data (a Copy into a table without a pre-copy script or a MERGE), rerunning only the failed step can leave duplicates or gaps. Check that the pipeline is idempotent before you rerun, and rerun the whole pipeline when in doubt.

#Rerun every failed run of one pipeline

$failed = Get-AzDataFactoryV2PipelineRun -ResourceGroupName $rg -DataFactoryName $df `
  -LastUpdatedAfter (Get-Date).AddHours(-12) -LastUpdatedBefore (Get-Date) |
  Where-Object { $_.PipelineName -eq 'pl_load_sales' -and $_.Status -eq 'Failed' }

foreach ($r in $failed) {
  $new = Invoke-AzDataFactoryV2Pipeline -ResourceGroupName $rg -DataFactoryName $df `
    -PipelineName $r.PipelineName -ReferencePipelineRunId $r.RunId -IsRecovery -StartFromFailure
  "{0} -> {1}" -f $r.RunId, $new
}

Run it for one pipeline at a time. A loop across every failed pipeline can push a self-hosted integration runtime past its concurrent jobs limit and turn a few failures into a queue.

#Cancel stuck runs

az datafactory pipeline-run cancel -g $RG --factory-name $DF --run-id <run-id> --is-recursive true

Stop-AzDataFactoryV2PipelineRun -ResourceGroupName $rg -DataFactoryName $df -PipelineRunId <run-id>

--is-recursive true also cancels child pipelines started by Execute Pipeline activities. Without it, the children keep running. Cancelling stops ADF waiting on the activity; work already submitted elsewhere (a Databricks job, a stored procedure) may keep running, so check and stop it at the target too.

Find everything still InProgress or Queued for more than three hours:

ADFPipelineRun
| where TimeGenerated > ago(1d)
| summarize arg_max(TimeGenerated, Status, Start) by RunId, PipelineName
| where Status in ("InProgress", "Queued") and Start < ago(3h)

#Pause and resume triggers

Before a maintenance window, stop triggers rather than disabling pipelines.

# What is running on a schedule right now
az datafactory trigger list -g $RG --factory-name $DF \
  --query "[].{name:name, state:properties.runtimeState, type:properties.type}" -o table

az datafactory trigger stop  -g $RG --factory-name $DF --name tr_daily_0200
az datafactory trigger start -g $RG --factory-name $DF --name tr_daily_0200

# Stop every started trigger and remember which ones they were
$started = Get-AzDataFactoryV2Trigger -ResourceGroupName $rg -DataFactoryName $df |
  Where-Object RuntimeState -eq 'Started'
$started.Name | Set-Content .\started-triggers.txt
$started | ForEach-Object { Stop-AzDataFactoryV2Trigger -ResourceGroupName $rg -DataFactoryName $df -Name $_.Name -Force }

# After the window: start only those
Get-Content .\started-triggers.txt | ForEach-Object {
  Start-AzDataFactoryV2Trigger -ResourceGroupName $rg -DataFactoryName $df -Name $_ -Force }

Caution

Saving the list matters. Starting every trigger after maintenance also starts triggers that were deliberately stopped, such as a backfill trigger or one for a retired source, and can load data twice.

Schedule trigger runs missed while stopped are not replayed. Tumbling window triggers do catch up on missed windows when restarted, which can start many runs at once; check maxConcurrency on the trigger first.

#Integration runtime health

az datafactory integration-runtime get-status -g $RG --factory-name $DF --name shir-prod \
  --query "properties.{state:state, nodes:nodes[].{node:nodeName, status:status, version:version}}"

Get-AzDataFactoryV2IntegrationRuntime -ResourceGroupName $rg -DataFactoryName $df -Name shir-prod -Status

A State other than Online, or a node in Offline or Limited, explains most copy failures from on-premises sources.

#Quick symptom map

#Prevent it next time

• Send PipelineRuns, ActivityRuns and TriggerRuns to Log Analytics for every factory, and alert on the "not succeeded since yesterday" query rather than only on failures.
• Keep the trigger stop/start script in the repo and use it in every change window.
• Design pipelines to be idempotent so that "rerun from failure" is always safe.
• Record the run IDs of any manual rerun in the incident ticket.