Home / Databricks on Azure

Databricks on AzureRunbookIntermediate

Sev1: Databricks jobs and pipelines failing auth — service principal secret or token expired

Restore ADF, CI/CD and external tools that call Azure Databricks after an Entra ID client secret, Databricks OAuth secret or PAT expired. Identify which credential, rotate it, update every consumer and move to managed identity or federation.

Deepak Bhardwaj3 min read

Applies toAzure Databricks, Entra ID service principals, Databricks OAuth M2M secrets, personal access tokens, Azure Data Factory linked services

Symptoms

  • ADF Databricks activities fail with 401/403 or Invalid access token.
  • CI/CD (databricks bundle deploy, Terraform) fails with authentication errors.
  • AADSTS7000222 (Entra secret expired) or invalid_client errors.

1. Which credential?

Caller authenticates with Expires Rotate in
Entra ID client secret → Entra token App registration secret end date Entra ID
Databricks OAuth secret (M2M) for a service principal Secret lifetime set at creation Databricks account/workspace
Personal access token (PAT) Token lifetime / max lifetime policy Workspace user settings or admin token management
ADF managed identity Doesn't expire Check permissions instead 
# PATs in the workspace (admin view)
databricks token-management list --output json | jq -r '.[] | [.token_id, .created_by_username, .comment, (.expiry_time/1000|todate)] | @tsv'

2. Rotate

Entra client secret:

az ad app credential reset --id <appId> --append --display-name "dbx-2026-10" --years 1 --query password -o tsv

Databricks OAuth secret for a service principal (account console → Service principals → Secrets → Generate secret, or the account API). Keep the old secret until consumers are updated, then delete it.

PAT (for a service principal, created on its behalf by an admin):

databricks token-management create-obo-token <sp-application-id> --lifetime-seconds 7776000 --comment "adf-prod-2026-10"

Put new values straight into Key Vault:

az keyvault secret set --vault-name kv-data --name dbx-sp-secret --value "<value>"

3. Update consumers

Consumer Where
ADF linked service Key Vault reference (preferred) or the linked service auth settings; re-publish
GitHub Actions / Azure DevOps Pipeline secrets / variable groups
Terraform ARM_CLIENT_SECRET or DATABRICKS_CLIENT_SECRET
Power BI / external BI tools Data source credentials

Run one pipeline end to end before declaring recovery, then rerun failed ADF runs from the failed activity.

4. Move to credentials that don't expire

  • ADF → Databricks: use the factory's managed identity in the Databricks linked service. Add the managed identity to the workspace (as a service principal) with permissions on the clusters/jobs it runs.
  • GitHub Actions / Azure DevOps → Databricks: workload identity federation (OIDC) with an Entra service principal, or Databricks OAuth token federation where supported — no stored secret.
  • People: OAuth U2M (databricks auth login) instead of PATs.

Then restrict PATs: workspace admin settings → set a maximum token lifetime, or disable PATs for users who don't need them.

Prevent it next time

  • Alert 30 days before any secret used by production expires (Entra app credentials, Databricks OAuth secrets, PATs).
  • One credential per consumer, so rotation affects one thing at a time.
  • All secrets in Key Vault, never in pipeline YAML or notebooks.

Related fixes