Calling Azure services from Databricks without secrets: Unity Catalog service credentials and access connectors

#The goal

Your notebooks and jobs call Azure services that are not storage: Key Vault, Event Hubs, Service Bus, Cosmos DB, Azure OpenAI. Today they authenticate with a service principal client secret stored in a secret scope. The secret expires, someone copies it into a notebook, and every rotation is a small outage.

A Unity Catalog service credential wraps the managed identity of an Azure Databricks access connector. Code asks Unity Catalog for a credential object, passes it to an Azure SDK client, and never sees a secret. Access is governed by Unity Catalog grants and audited like any other securable.

Note

Service credentials are for calling external services. For reading and writing ADLS data, keep using storage credentials and external locations.

#Requirements

Feature support moves quickly. Check the current Azure Databricks documentation for your runtime before you plan around UDF support.

#Step 1: Create the access connector and grant it Azure roles

Use a dedicated access connector per purpose, for example one per environment for "integration" calls. Don't reuse the connector behind your storage credential.

RG=rg-dbx-prod
LOC=westeurope

# The access connector commands come from the Azure CLI "databricks" extension
az extension add --name databricks --upgrade

az databricks access-connector create \
  --resource-group $RG \
  --name ac-dbx-prod-integration \
  --location $LOC \
  --identity-type SystemAssigned

PRINCIPAL_ID=$(az databricks access-connector show \
  --resource-group $RG --name ac-dbx-prod-integration \
  --query identity.principalId -o tsv)

CONNECTOR_ID=$(az databricks access-connector show \
  --resource-group $RG --name ac-dbx-prod-integration \
  --query id -o tsv)
echo $CONNECTOR_ID

Then grant the identity only the data-plane roles it needs on each target resource:

KV_ID=$(az keyvault show --name kv-app-prod --query id -o tsv)
az role assignment create \
  --assignee-object-id $PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope $KV_ID

EH_ID=$(az eventhubs namespace show -g $RG -n evh-app-prod --query id -o tsv)
az role assignment create \
  --assignee-object-id $PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "Azure Event Hubs Data Sender" \
  --scope $EH_ID

Tip

Key Vault RBAC roles only work when the vault uses the Azure RBAC permission model. If the vault still uses access policies, add an access policy for the connector's principal instead, or migrate the vault to RBAC.

Role assignments can take several minutes to take effect. A 403 right after assignment is usually propagation delay, not a wrong role.

#Step 2: Create the service credential in Unity Catalog

In Catalog Explorer: Catalog > Connect > Credentials > Create credential > Service Credential.

1. Name it after its purpose, for example sc_prod_integration.
2. Paste the access connector resource ID ($CONNECTOR_ID from above): /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Databricks/accessConnectors/<connector-name>
3. If you use a user-assigned managed identity on the connector, also paste its resource ID.
4. Create it.

Prefer managed identity over a service principal here. A service principal can't delegate CREATE SERVICE CREDENTIAL, and a service principal needs the account admin role to create a credential that uses a managed identity. Plan for a human admin, or your Terraform identity with the right role, to create these.

#Step 3: Grant access and bind to workspaces

Grant to groups, not individual users:

GRANT ACCESS ON SERVICE CREDENTIAL sc_prod_integration TO `grp-data-engineers-prod`;
GRANT ACCESS ON SERVICE CREDENTIAL sc_prod_integration TO `sp-prod-jobs`;

-- Remove access
REVOKE ACCESS ON SERVICE CREDENTIAL sc_prod_integration FROM `grp-data-engineers-prod`;

By default a service credential is usable from every workspace attached to the metastore. That includes dev and test, which is rarely what you want for a production identity. To restrict it, open the credential, go to Workspaces, clear All workspaces have access, and click Assign to workspaces. You need to be a metastore admin or the credential owner to do this.

Warning

If you skip workspace binding, any user in a dev workspace with ACCESS on the credential can act as your production managed identity. Bind production credentials to production workspaces only.

#Step 4: Use it from a notebook or job

dbutils.credentials.getServiceCredentialsProvider returns a credential object that Azure SDK clients accept wherever they take a credential argument.

%pip install azure-keyvault-secrets azure-eventhub

from azure.keyvault.secrets import SecretClient

credential = dbutils.credentials.getServiceCredentialsProvider("sc_prod_integration")

kv = SecretClient(vault_url="https://kv-app-prod.vault.azure.net/", credential=credential)
api_key = kv.get_secret("partner-api-key").value

Event Hubs producer with the same credential:

from azure.eventhub import EventHubProducerClient, EventData

credential = dbutils.credentials.getServiceCredentialsProvider("sc_prod_integration")

producer = EventHubProducerClient(
    fully_qualified_namespace="evh-app-prod.servicebus.windows.net",
    eventhub_name="orders-processed",
    credential=credential,
)
with producer:
    batch = producer.create_batch()
    batch.add(EventData('{"run_id": "2026-11-02", "status": "done"}'))
    producer.send_batch(batch)

Take the credential name from a job parameter or widget rather than hard-coding it. The same notebook can then run against sc_dev_integration and sc_prod_integration.

#Inside UDFs

dbutils isn't available inside UDF execution. In UDFs use databricks.service_credentials.getServiceCredentialsProvider() instead. Which UDF types support it depends on the runtime, access mode and serverless environment version. Check the current docs before you rely on it from a SQL warehouse.

#Default credential

You can set an environment variable on a classic cluster under Advanced > Spark > Environment variables:

DATABRICKS_DEFAULT_SERVICE_CREDENTIAL_NAME=sc_prod_integration

DefaultAzureCredential() then picks it up without a name in code. Serverless compute and SQL warehouses don't support this, and Databricks doesn't recommend it because the code becomes less portable. Pass the name explicitly.

#Troubleshooting

Check what the identity can actually do:

az role assignment list --assignee $PRINCIPAL_ID --all \
  --query "[].{role:roleDefinitionName, scope:scope}" -o table

#Migrating off client secrets

1. Inventory notebooks that read service principal secrets: search the repo for dbutils.secrets.get and ClientSecretCredential.
2. Grant the access connector the same Azure roles the old service principal had, scoped no wider.
3. Swap ClientSecretCredential(...) for getServiceCredentialsProvider(...). The client calls stay the same.
4. Run the job once on the new path, then remove the old app's role assignments and delete its secret.

#Prevent it next time

• One service credential per environment and purpose, bound to that environment's workspaces only.
• Grant ACCESS to groups and service principals that run jobs, never to all account users.
• Manage the access connector and its role assignments as code, next to the Unity Catalog grants.
• Review grants and the audit log for service credential use in the same cadence as storage credentials.
• Add a CI check that fails on new ClientSecretCredential usage in notebooks.