Sev1: Databricks jobs fail reading secrets — Key Vault-backed secret scope broken by firewall or permission change

#Symptoms

• Notebook or job fails at dbutils.secrets.get(scope="kv-prod", key="sql-password").
• Errors mention Forbidden, Secret does not exist, or that the Key Vault is not accessible.
• Many jobs fail at once, across clusters.

#1. Confirm the scope configuration

databricks secrets list-scopes
databricks secrets list-secrets kv-prod

list-scopes shows the backend type and, for Azure Key Vault scopes, the vault resource ID and DNS name. If list-secrets fails too, the problem is between Databricks and the vault, not in your code.

#2. Network: vault firewall

az keyvault show -n kv-dbx-prod \
  --query "{public:properties.publicNetworkAccess, default:properties.networkAcls.defaultAction, bypass:properties.networkAcls.bypass, ips:properties.networkAcls.ipRules, vnets:properties.networkAcls.virtualNetworkRules}"

Key Vault-backed scopes are read by the Azure Databricks control plane (not your cluster). With the firewall enabled you need Allow trusted Microsoft services (bypass: AzureServices) so the AzureDatabricks first-party app can reach the vault, unless your setup documents another path:

az keyvault update -n kv-dbx-prod --bypass AzureServices

If someone set --bypass None or disabled public access entirely during a hardening change, secret scopes break immediately. Check the activity log:

az monitor activity-log list --resource-id $(az keyvault show -n kv-dbx-prod --query id -o tsv) --offset 24h \
  --query "[].{t:eventTimestamp, op:operationName.value, who:caller}" -o table

#3. Permissions: access policies or RBAC

The AzureDatabricks enterprise application needs read access to secrets.

• Access policy model: a policy with Get and List secret permissions for the AzureDatabricks application.

az keyvault set-policy -n kv-dbx-prod --spn 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d \
  --secret-permissions get list

(2ff814a6-... is the well-known application ID of AzureDatabricks; verify it in your tenant's enterprise applications.)

• Azure RBAC model: Key Vault Secrets User for the AzureDatabricks service principal at vault scope.

Switching the vault from access policies to RBAC (or back) without recreating the grants is a common cause of this outage.

#4. The secret itself

az keyvault secret show --vault-name kv-dbx-prod --name sql-password --query "{enabled:attributes.enabled, expires:attributes.expires}"
az keyvault secret list-deleted --vault-name kv-dbx-prod -o table

• Disabled or expired secrets fail reads.
• A deleted secret can be recovered: az keyvault secret recover --vault-name kv-dbx-prod --name sql-password.

#5. Verify

v = dbutils.secrets.get(scope="kv-prod", key="sql-password")
print(len(v))   # never print the value

Rerun the failed jobs.

Tip

Consider Unity Catalog service credentials or managed identity connections for Azure resources so fewer jobs depend on stored passwords at all.

#Prevent it next time

• Document the Key Vault settings Databricks depends on (trusted services bypass, AzureDatabricks grants) in the vault's IaC.
• Keep one access model per vault and migrate deliberately.
• Alert on Key Vault Forbidden responses (diagnostic logs, AzureDiagnostics with ResultSignature == "Forbidden").
• Track secret expiry dates and rotate before they expire.