Azure Databricks secret scopes cheat sheet: Key Vault-backed scopes, ACLs and dbutils.secrets

A job fails because a password, API key or JDBC credential can't be read, or someone pasted a key straight into a notebook and you need to move it somewhere safe. This page has the commands for both cases, plus a table of the errors you'll see most.

Note

For cloud storage, don't use secrets at all. Use a Unity Catalog storage credential with an access connector instead. Secrets are for things Unity Catalog can't broker: database passwords, third-party API keys, service principal secrets for external systems.

#Scope types in one table

Warning

Secret ACLs apply to the scope, not to individual keys. Anyone with READ on a Key Vault-backed scope can read every secret in that vault. Use one vault per application or team, not one vault for everything.

#Set up a Key Vault-backed scope

#1. Prepare the vault

Microsoft's documentation states that Key Vault-backed scopes support the vault access policy permission model, not Azure RBAC. Check the model before you start:

az keyvault show --name kv-app-prod \
  --query "{rbac:properties.enableRbacAuthorization, uri:properties.vaultUri, id:id, publicAccess:properties.publicNetworkAccess}" -o json

If rbac is true, switch the vault to Access configuration → Vault access policy before you create the scope. Changing the model affects anyone who uses RBAC on that vault today, so agree on it with the vault owner first.

If the vault firewall is on, tick Allow trusted Microsoft services to bypass this firewall:

az keyvault update --name kv-app-prod --bypass AzureServices

You need Contributor, Key Vault Contributor or Owner on the vault to create the scope, even if Databricks already has access.

#2. Create the scope

UI: open https://<workspace-url>#secrets/createScope (case-sensitive, capital S). Then enter the scope name, the DNS name (the vault URI) and the resource ID.

CLI (0.205 and above), using the request body form:

databricks secrets create-scope --json '{
  "scope": "kv-app-prod",
  "scope_backend_type": "AZURE_KEYVAULT",
  "backend_azure_keyvault": {
    "resource_id": "/subscriptions/<sub-id>/resourceGroups/rg-app-prod/providers/Microsoft.KeyVault/vaults/kv-app-prod",
    "dns_name": "https://kv-app-prod.vault.azure.net/"
  }
}'

Authenticate the CLI with a Microsoft Entra ID identity (for example auth_type = azure-cli in your profile). Personal access tokens are not accepted for creating Key Vault-backed scopes. Creating the scope grants the AzureDatabricks application Get and List on secrets through an access policy.

#3. Add secrets in Key Vault

az keyvault secret set --vault-name kv-app-prod --name sqldb-etl-password --value '<password>'

Key Vault secret names allow only letters, digits and dashes. Use sqldb-etl-password, not sqldb_etl_password.

#Databricks-backed scopes

databricks secrets create-scope app-dev
databricks secrets put-secret app-dev api-token --string-value '<token>'

# multi-line values (certificates, JSON keys) from stdin
cat sa-key.json | databricks secrets put-secret app-dev gcp-sa-key

Remove the source file afterwards and make sure the value isn't left in your shell history.

#Everyday CLI commands

Caution

delete-scope removes the scope and every ACL on it immediately, and every job that reads from it breaks. For Key Vault-backed scopes the secrets stay in the vault, but you'll have to recreate the scope and its ACLs.

#Permissions

• Principals: users by email, groups by name, service principals by application ID.
• put-acl on a principal that already has an ACL overwrites it. It does not add to it.
• The scope creator gets MANAGE by default. Workspace admins can always manage scopes.
• Grant to groups and job service principals, not individuals:

databricks secrets put-acl kv-app-prod 9f3c2a1e-0000-0000-0000-000000000000 READ   # job SP
databricks secrets put-acl kv-app-prod data-engineers READ
databricks secrets put-acl kv-app-prod platform-admins MANAGE

#Reading secrets

#Python notebooks and jobs

dbutils.secrets.listScopes()
dbutils.secrets.list("kv-app-prod")          # key names only

pwd = dbutils.secrets.get(scope="kv-app-prod", key="sqldb-etl-password")

df = (spark.read.format("jdbc")
      .option("url", "jdbc:sqlserver://sql-prod.database.windows.net:1433;database=sales")
      .option("dbtable", "dbo.orders")
      .option("user", "etl_reader")
      .option("password", pwd)
      .load())

Printing pwd shows [REDACTED]. Redaction only catches the literal value, so a reversed or base64-encoded string will print in clear. ACLs are the real control, not redaction.

#SQL

SELECT secret('kv-app-prod', 'api-token') IS NOT NULL AS token_readable;

Databricks blocks DML that would write a raw secret into a table. Wrapping the value in a cryptographic function such as aes_encrypt() is allowed.

#Cluster Spark config and environment variables

spark.myapp.password {{secrets/kv-app-prod/sqldb-etl-password}}

SQLDB_PASSWORD={{secrets/kv-app-prod/sqldb-etl-password}}

The cluster owner needs READ on the scope. The value is resolved at cluster start, so restart the cluster after rotating a secret.

#Errors and fixes

#Prevent it next time

• One scope per application and environment (kv-sales-prod, kv-sales-dev), with the vault split the same way.
• Grant ACLs to groups and job service principals only. Review them with list-acls in your access reviews.
• Rotate in Key Vault on a schedule, and restart long-running clusters that read secrets from Spark config.
• Search repos and notebooks for keys and connection strings, and move anything you find into a scope.
• Newer workspaces can also store secrets as Unity Catalog securables. Check whether that feature is available in your region before you standardise on scopes.