Sev1: Data Factory Databricks activities all failing — linked service auth, managed identity and cluster settings

#Symptoms

• Databricks Notebook, Jar, Python or Job activities fail in many pipelines at once.
• Errors such as 401 Unauthorized, 403 Forbidden, Invalid access token, The cluster ... does not exist, or cluster creation failures.

#1. Read the activity error

In ADF monitor, open the failed activity's output. Note the errorCode and the Databricks runPageUrl if a run was created.

#2. Token-based linked service

Personal access tokens expire or get revoked (for example by a token cleanup after a leak). Move away from PATs:

• Best: managed identity authentication on the linked service (Step 3).
• If you must keep a token, use a token for a service principal, stored in Key Vault, and reference it from the linked service.

#3. Managed identity authentication

The Data Factory's managed identity must be a member of the workspace, with permission to run the jobs or use clusters.

Add it to the workspace (workspace admin, Databricks CLI):

databricks service-principals create --json '{
  "application_id": "<adf-managed-identity-app-id>",
  "display_name": "adf-prod"
}'

Then grant what the activities need:

• Can Attach To / Can Restart on an existing interactive cluster, or allowance to create clusters (or a cluster policy permission) for new job clusters.
• Can Manage Run on Databricks jobs triggered by the Job activity.
• Unity Catalog privileges on the data the notebooks read and write, since the identity runs the code.

Microsoft's setup for managed identity auth (linked service pointing at the workspace resource ID) grants the factory's identity the Contributor role on the workspace resource, which makes it a workspace admin on first sign-in. If someone removed that role assignment during an RBAC cleanup, every activity fails with 403. Check it:

az role assignment list --scope <databricks-workspace-resource-id> \
  --query "[].{who:principalName, role:roleDefinitionName}" -o table

If you prefer least privilege, add the identity as a normal workspace service principal with only the permissions above, and check the linked service documentation for what your auth mode requires.

#4. Cluster settings in the linked service

• Existing cluster ID: if the cluster was deleted or recreated, update the ID. Prefer new job clusters or instance pools for production; they don't drift.
• New job cluster: node type, runtime version and policy must still be valid. A retired runtime version or a new cluster policy that forbids the node type makes creation fail:

databricks clusters spark-versions | head
databricks cluster-policies list

• Instance pool: check the pool has capacity (max_capacity) and idle instances.

#5. Network path

If the workspace uses Private Link with public access disabled, ADF's Azure integration runtime can't reach it. Use a managed virtual network integration runtime with a managed private endpoint to the workspace (databricks_ui_api), approved on the workspace side.

az network private-endpoint-connection list --id <databricks-workspace-resource-id> -o table

#6. Rerun failed pipelines

After the fix, rerun from the failed activity. Make sure notebooks are idempotent, since some may have partly run before failing.

Tip

Consider the Databricks Job activity (runs an existing Databricks job) instead of notebook activities. The job's own cluster, retries and permissions are then managed in Databricks, and ADF only triggers it.

#Prevent it next time

• Managed identity auth on all Databricks linked services; no personal tokens.
• Job clusters or pools with a fixed policy instead of interactive cluster IDs.
• Alerts on ADF activity failures grouped by linked service.
• Include ADF in the change review for workspace networking and policy changes.