Home / Data Platform / Data Factory

Data PlatformTroubleshootingIntermediate

Azure Data Factory mapping data flow failures - cluster start errors, out of memory, broadcast timeouts and sink errors

Mapping data flows run on managed Spark, so failures look like Spark failures. Find the error code in the activity output, then fix the cluster size, joins/broadcast, partitioning or sink settings.

Deepak Bhardwaj5 min read

Applies toAzure Data Factory and Synapse pipelines mapping data flows; Azure Integration Runtime with data flow compute; Az PowerShell and Log Analytics

Get the real error

The pipeline view shows a short message. The full error is in the activity output:

$runs = Get-AzDataFactoryV2ActivityRun -ResourceGroupName rg-data -DataFactoryName adf-prod `
  -PipelineRunId <run-id> -RunStartedAfter (Get-Date).AddDays(-1) -RunStartedBefore (Get-Date)
$runs | Where-Object ActivityType -eq "ExecuteDataFlow" |
  Select-Object ActivityName, Status, @{n='Error';e={$_.Error.message}} | Format-List

If diagnostic settings send logs to Log Analytics (resource-specific tables):

ADFActivityRun
| where TimeGenerated > ago(1d)
| where ActivityType == "ExecuteDataFlow" and Status == "Failed"
| project TimeGenerated, PipelineName, ActivityName, ErrorCode, ErrorMessage
| order by TimeGenerated desc

Common error codes start with DF- (for example DF-Executor-... or DF-Cluster-...). Search the code in the Microsoft "Troubleshoot mapping data flows" page for the exact meaning.

Error → cause → fix

Error pattern Cause Fix
Cluster failed to start / timed out Capacity or quota in region, or managed VNet startup Retry; check IR region and managed VNet settings; use a TTL to keep clusters warm
OutOfMemoryError, "Executor lost", "Container killed" Data skew or too-small compute Larger core count / memory-optimised, fix skew, reduce wide transformations
BroadcastTimeout / broadcast join failure Large side of a join was broadcast Set broadcast to Off or Fixed on the small side only
Sink "column does not exist" / type conversion Schema drift or mapping mismatch Enable schema drift deliberately, or fix the mapping and data types
Sink deadlocks / timeouts in Azure SQL Too many parallel writers Lower partitions on the sink, use batch size, or stage and load
"Source file not found" with wildcards Path/wildcard wrong or file moved Check the folder path and wildcard on the source
Debug works, pipeline fails Different parameters, IR or data volume Compare run parameters and IR used

Out of memory and skew

Open the failed run in the monitoring view of the data flow. It shows rows and time per transformation and per partition.

  • One partition far larger than the others is skew. Typical causes are joins or aggregates on a key with a dominant value (NULL, "UNKNOWN", one big customer).
  • Fixes:
    • Filter or handle NULL keys separately before the join.
    • Add a salt column to spread a hot key, then aggregate in two steps.
    • Set Optimize > Partitioning to Hash on a well-distributed column, or round robin before heavy steps.
  • Increase compute only after you've checked for skew. Use the Data flow properties of the activity (compute size, or core count and compute type).

Broadcast joins

Data flows broadcast the smaller side of joins, lookups and exists automatically. If the "small" side is actually big, the broadcast times out or exhausts memory.

In the Join, Lookup or Exists transformation, open Optimize > Broadcast:

  • Auto: the engine decides. This is the default and can guess wrong.
  • Fixed: broadcast only the side you choose, which must be genuinely small (dimension tables).
  • Off: no broadcast. Slower, but safe for large-to-large joins.

Sink problems

Azure SQL / Synapse sinks

  • Set the sink partitioning to a modest count. Each partition opens its own connection.
  • For large loads, enable staging (PolyBase/COPY for Synapse) and write to a staging table, then swap or merge.
  • Pre-create the target table with the right types. Letting the data flow create it gives generic types.

Delta / Parquet sinks

  • Too many output files: set partitioning to Single partition for small outputs, or a fixed count for large ones.
  • Concurrent writes to the same Delta table from two runs can fail. Serialise them with pipeline concurrency = 1.

Cluster start time and cost

Each data flow activity starts a Spark cluster unless an IR with time to live (TTL) keeps one warm.

"typeProperties": {
  "computeProperties": { "location": "AutoResolve",
    "dataFlowProperties": { "computeType": "General", "coreCount": 16, "timeToLive": 10 } }
}
  • TTL saves start time for sequential data flows in the same pipeline, but you pay for the idle minutes.
  • Data flows running in parallel each need their own cluster. TTL reuse only helps sequential runs.

Tip

For heavy or complex transformations that run often, consider moving them to Databricks or Fabric notebooks. Data flows are best for moderate, visual transformations.

Prevent it next time

  • Profile source data for NULL and hot keys before designing joins.
  • Set broadcast explicitly on joins with large inputs.
  • Alert on failed data flow activities in Log Analytics.
  • Keep a test dataset at production scale for debug runs before release.

Checklist

  • Full error code read from activity output or logs
  • Partition skew checked in the monitoring view
  • Broadcast set explicitly on large joins
  • Compute sized after fixing skew, not before
  • Sink partitions and staging tuned
  • TTL used only where it pays off

Related fixes