Home / Data Platform / Data Factory

Data PlatformHow-toIntermediate

High availability for the ADF self-hosted integration runtime: multi-node setup

Add nodes to a self-hosted integration runtime for HA and scale-out, configure node-to-node communication, plan patching and updates, and monitor node health.

Deepak Bhardwaj2 min read

Applies toAzure Data Factory / Synapse self-hosted integration runtime (up to 4 nodes)

How multi-node SHIR works

  • One logical SHIR in ADF; up to four nodes (VMs) registered with the same key.
  • Nodes work active-active: activities are dispatched to available nodes.
  • If a node goes down, new activities go to the remaining nodes. Activities running on the failed node fail and need retries to rerun.
  • One node acts as dispatcher/worker; if it fails, another takes over the dispatcher role.

1. Add a second node

On the new VM (same SHIR version):

msiexec /i IntegrationRuntime.msi /quiet
& "C:\Program Files\Microsoft Integration Runtime\5.0\Shared\dmgcmd.exe" -RegisterNewNode "<same-authentication-key>" "shir-node-2"
& "C:\Program Files\Microsoft Integration Runtime\5.0\Shared\dmgcmd.exe" -EnableRemoteAccess 8060

Remote access (port 8060, TLS with a certificate trusted by all nodes) lets nodes sync credentials. Use a certificate from your PKI for production.

2. Placement

  • Nodes in different hosts/racks, or different availability zones if the SHIR runs in Azure.
  • Same network reach to all data sources (each node must reach every source).
  • Same Java runtime and drivers (ODBC drivers, SAP connectors) installed on every node.

3. Pipeline settings

  • Activity retries (for example 2 retries, 60 seconds interval) on copy and lookup activities, because in-flight work on a failed node fails.
  • Idempotent sinks (truncate-and-load into staging, or upserts), so reruns are safe.

4. Patching and updates

  • Patch nodes one at a time; check the other nodes are Online first.
  • Auto-update applies to all nodes; with "manual" updates, upgrade nodes one by one to the same version.
  • Before rebooting a node, use the SHIR configuration manager to stop accepting new work, or schedule reboots outside pipeline windows.

5. Monitoring

Get-AzDataFactoryV2IntegrationRuntimeMetric -ResourceGroupName rg-adf -DataFactoryName adf-prod -Name shir-prod
Get-AzDataFactoryV2IntegrationRuntime -ResourceGroupName rg-adf -DataFactoryName adf-prod -Name shir-prod -Status

Alert on node status Offline or Limited and on high CPU/memory per node.

6. DR

A SHIR is tied to one data factory. For DR:

  • Keep the factory definition in Git and redeploy to a DR factory in another region.
  • Install SHIR nodes for the DR factory in the DR site (or link a shared SHIR from another factory).
  • Test a DR pipeline run as part of drills.

Tip

"Limited" status often means nodes can't talk to each other on port 8060 (certificate or firewall). Fix that before trusting the HA setup.

Related fixes