Home / Databricks on Azure

Databricks on AzureHow-toIntermediate

Transformation: replace Databricks personal access tokens with OAuth and managed identities

Move users, CI/CD, ADF and external tools off long-lived PATs. Use OAuth U2M for people, service principals with OAuth M2M or Entra federation for pipelines, ADF managed identity, then restrict and expire PATs.

Deepak Bhardwaj3 min read

Applies toAzure Databricks, Databricks CLI and SDKs (unified authentication), Entra ID service principals, Azure Data Factory, GitHub Actions / Azure DevOps

Inventory

databricks token-management list --output json | \
  jq -r '.[] | [.created_by_username, .comment, (.creation_time/1000|todate), (if .expiry_time == -1 then "never" else (.expiry_time/1000|todate) end)] | @tsv' | sort

Group tokens by owner: people, service principals, and "people's tokens used by automation" (the riskiest kind).

1. People: OAuth U2M

databricks auth login --host https://adb-1234567890123456.7.azuredatabricks.net --profile prod
databricks clusters list --profile prod

Tokens are short-lived and refreshed automatically. IDEs (VS Code extension) and the SDKs use the same profiles. Tools that connect via ODBC/JDBC (Power BI, Tableau, DBeaver) support OAuth/Entra ID sign-in too — switch their connection auth type.

2. CI/CD: service principal without secrets

GitHub Actions / Azure DevOps with Entra workload identity federation:

  1. Entra service principal with a federated credential for the repo/environment.
  2. Add the service principal to the workspace (account console) with the permissions it needs.
  3. Authenticate with the Azure CLI login step, then the Databricks CLI uses azure-cli auth:
- uses: azure/login@v2
  with: { client-id: ${{ vars.AZURE_CLIENT_ID }}, tenant-id: ${{ vars.AZURE_TENANT_ID }}, allow-no-subscriptions: true }
- run: databricks bundle deploy -t prod
  env:
    DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
    DATABRICKS_AUTH_TYPE: azure-cli

Databricks also supports OAuth token federation directly with GitHub's OIDC tokens; check availability for your account.

Where federation isn't possible: a Databricks service principal with an OAuth M2M secret (DATABRICKS_CLIENT_ID / DATABRICKS_CLIENT_SECRET) — short-lived access tokens, one rotatable secret.

3. Azure Data Factory

Use the factory's managed identity in the Azure Databricks linked service:

  1. Add the ADF managed identity to the workspace as a service principal (by its application ID).
  2. Grant it permissions on the clusters/policies/jobs it uses (CAN_MANAGE_RUN on jobs, CAN_ATTACH_TO/CAN_RESTART on clusters or CAN_USE on a policy).
  3. Switch the linked service authentication to Managed service identity and remove the token.

4. Jobs running as people

Change run_as of production jobs to a service principal so they don't depend on a person's account or token:

resources:
  jobs:
    nightly_sales:
      run_as:
        service_principal_name: "8a7c1e2f-0000-1111-2222-333344445555"

5. Restrict PATs

  • Workspace settings: set a maximum lifetime for new tokens.
  • Token permissions: only a small group (or nobody) can create PATs.
  • Delete old tokens after their replacement is proven, starting with "never expires" ones.

Tip

Do one integration end to end per day and delete its PAT the same day. Leaving the old token "just in case" is how migrations stall at 90%.

Checklist

  • PAT inventory grouped by owner/integration
  • Users on OAuth U2M; BI tools on OAuth/Entra
  • CI/CD on federation or M2M OAuth; ADF on managed identity
  • Jobs run as service principals
  • PAT creation restricted; old tokens deleted

Related fixes