Home / Data Platform / Data Factory

Data PlatformRunbookIntermediate

Sev1: Data Factory self-hosted integration runtime offline — copy activities from on-premises failing

Every pipeline that reads on-premises sources fails because the self-hosted integration runtime is offline or limited. Check node status, service, connectivity, certificates and capacity, then rerun.

Deepak Bhardwaj3 min read

Applies toAzure Data Factory and Synapse pipelines, self-hosted integration runtime 5.x, Az.DataFactory PowerShell

Symptoms

  • Copy and lookup activities fail with messages that the integration runtime is offline, unavailable or has no available nodes.
  • The SHIR shows Offline or Limited in the ADF monitor.
  • Started after server patching, a password change, a proxy change or an expired certificate.

1. Check status from Azure

Get-AzDataFactoryV2IntegrationRuntime -ResourceGroupName rg-data -DataFactoryName adf-prod `
  -Name shir-onprem -Status | Select-Object State, Nodes, Version, AutoUpdate

Get-AzDataFactoryV2IntegrationRuntimeNode -ResourceGroupName rg-data -DataFactoryName adf-prod `
  -IntegrationRuntimeName shir-onprem -Name <node-name> | Select-Object Status, LastConnectTime, LastStartTime, ConcurrentJobsLimit
State Meaning
Offline No node can reach the service
Limited Some node problem, often node-to-node communication in a multi-node setup
NeedRegistration Node lost its registration
Online but jobs queue Capacity: concurrent jobs limit or CPU/memory

2. On the SHIR machine

Get-Service DIAHostService
Restart-Service DIAHostService
Get-WinEvent -LogName "Integration Runtime" -MaxEvents 30 | Format-List TimeCreated, LevelDisplayName, Message

If the service fails to start, check the service logon account. A domain account whose password changed breaks the service; update it in Services or switch to the default NT SERVICE\DIAHostService.

3. Connectivity to Azure

The SHIR needs outbound HTTPS (443) to the Data Factory and Service Bus endpoints for your region. Use the diagnostics tool in the Microsoft Integration Runtime Configuration Manager (Diagnostics > Test), or:

Test-NetConnection -ComputerName <your-region>.frontend.clouddatahub.net -Port 443
Test-NetConnection -ComputerName login.microsoftonline.com -Port 443

If traffic goes through a proxy, check the proxy settings in Configuration Manager (Settings > HTTP proxy). A proxy password change or new SSL inspection rule commonly breaks it.

4. Re-register a node

If a node shows NeedRegistration or the machine was rebuilt:

$key = (Get-AzDataFactoryV2IntegrationRuntimeKey -ResourceGroupName rg-data `
  -DataFactoryName adf-prod -Name shir-onprem).AuthKey1
& "C:\Program Files\Microsoft Integration Runtime\5.0\Shared\dmgcmd.exe" -RegisterNewNode $key

(The path includes the installed major version; check your install folder.)

Warning

Regenerating the auth keys (New-AzDataFactoryV2IntegrationRuntimeKey) disconnects nodes registered with the old key. Only regenerate if a key leaked, and re-register every node immediately.

5. Multi-node: Limited state

Nodes talk to each other over TCP 8060. If it is blocked by a host firewall, or the nodes use a TLS certificate for intranet communication that expired, the runtime goes Limited. Check the certificate in Configuration Manager and the firewall rule.

6. Capacity problems

If the runtime is online but jobs queue or time out:

  • Check CPU and memory on nodes during the load window.
  • Raise the concurrent jobs limit if the machine has headroom:
Update-AzDataFactoryV2IntegrationRuntimeNode -ResourceGroupName rg-data -DataFactoryName adf-prod `
  -IntegrationRuntimeName shir-onprem -Name <node-name> -ConcurrentJobsLimit 16
  • Add a second node for both capacity and high availability.

7. Rerun failed pipelines

Rerun from the failed activity in the ADF monitor, in business-priority order, so on-premises sources are not flooded.

Prevent it next time

  • At least two SHIR nodes on separate machines, patched on different days.
  • Alerts on integration runtime status and on node LastConnectTime.
  • Keep auto-update on (or a monthly manual update) to stay on supported versions.
  • Document proxy and firewall rules the SHIR depends on with the network team.