Azure Data Factory self-hosted IR saturated: copy activities queueing, high CPU and memory, concurrent jobs limits and scale-out

#Symptoms

• In the monitoring view, copy activities show long queue time before transfer starts ("Time to first byte" or "Queue" in the activity details).
• The nightly load finishes later every week as more tables are added.
• SHIR nodes run at 90–100% CPU or are low on memory during the load window; some activities fail with timeouts or out-of-memory errors.

#Step 1: read the IR metrics

In Azure Monitor (Data Factory resource metrics):

PowerShell snapshot:

Get-AzDataFactoryV2IntegrationRuntimeMetric -ResourceGroupName rg-data `
  -DataFactoryName adf-prod -Name shir-onprem |
  Select-Object -ExpandProperty Nodes |
  Format-Table NodeName, AvailableMemoryInMB, CpuUtilization, ConcurrentJobsLimit, ConcurrentJobsRunning, SentBytes, ReceivedBytes

Get-AzDataFactoryV2IntegrationRuntime -ResourceGroupName rg-data -DataFactoryName adf-prod `
  -Name shir-onprem -Status | Select-Object State, Version, AutoUpdate, Nodes

#Step 2: identify the bottleneck

#Step 3: fix concurrency

Each node runs a limited number of concurrent jobs. The default is computed from cores and memory; you can raise it if CPU and memory have headroom:

Update-AzDataFactoryV2IntegrationRuntimeNode -ResourceGroupName rg-data -DataFactoryName adf-prod `
  -IntegrationRuntimeName shir-onprem -Name SHIRNODE01 -ConcurrentJobsLimit 24

Raising it on a CPU-bound node makes everything slower, not faster.

Also limit concurrency on the pipeline side so a single ForEach with 50 parallel copies doesn't flood the IR:

"typeProperties": { "isSequential": false, "batchCount": 10 }

#Step 4: fix CPU and memory

The SHIR does the heavy lifting for on-premises sources: reading, serialising, compressing and converting formats.

• Format conversions are expensive. Writing Parquet/ORC on the SHIR uses a lot of CPU and JVM memory. Consider landing data as compressed CSV/delimited text and converting in the cloud (Spark, Fabric, Databricks), or using staged copy.
• Java heap for Parquet/ORC: the SHIR uses a JRE for these formats; large writes need more heap. Set the _JAVA_OPTIONS system environment variable on the node (for example -Xms256m -Xmx16g) and restart the IR service; size it to the node's RAM.
• Parallel copies per activity: parallelCopies higher than the node can handle overloads it; set it explicitly instead of letting dozens of activities each choose a high default.
• Right-size the node: SHIR nodes for heavy loads need many cores and plenty of RAM, and fast local disk for temp files.

#Step 5: scale out

A SHIR can have up to four nodes (check the current limit). Install the IR on another machine and register it with the same authentication key:

Get-AzDataFactoryV2IntegrationRuntimeKey -ResourceGroupName rg-data -DataFactoryName adf-prod -Name shir-onprem
# On the new node, in the IR configuration manager or:
# & "C:\Program Files\Microsoft Integration Runtime\5.0\Shared\dmgcmd.exe" -RegisterNewNode "<key>" "SHIRNODE02"

Nodes should be similar in size and in network path to sources. Multi-node also gives high availability during patching.

For very large estates, split workloads across separate IRs (for example one per source system or one for Oracle, one for file shares) so a heavy job can't starve everything.

#Step 6: network and source checks

• Use the SHIR's diagnostic tools and Windows performance counters to check NIC throughput.
• Check source DB waits during extraction (long-running queries, locks, slow disks).
• For large tables, use partitioned reads (dynamic range or physical partitions) so several queries extract in parallel, balanced against source load.

"source": {
  "type": "SqlServerSource",
  "partitionOption": "DynamicRange",
  "partitionSettings": { "partitionColumnName": "OrderId" }
}

#Step 7: keep it healthy

• Turn on auto-update or patch on a schedule; old IR versions have known performance and memory bugs.
• Send IR event logs ("Integration Runtime" and "Connectors - Integration Runtime" in Event Viewer) to Log Analytics via Azure Monitor Agent.
• Alert on IntegrationRuntimeQueueDuration and CPU above thresholds during load windows.

Tip

If most data ends up in Azure anyway, ask whether the source can be reached from an Azure IR through a private endpoint or managed VNet. That moves heavy work off your SHIR nodes.

#Prevent it next time

• Capacity-plan SHIR nodes like any other server: CPU, memory and network at peak.
• Cap ForEach batch counts and parallelCopies explicitly.
• Land raw, convert in the cloud.
• Review queue duration weekly as table counts grow.