Sev1: every Unity Catalog table returns 403 after the Databricks access connector or its role was removed

#Symptoms

• Every query on Unity Catalog tables in a catalog (or the whole metastore) fails:
  • AbfsRestOperationException: Operation failed: "This request is not authorized to perform this operation using this permission.", 403, AuthorizationPermissionMismatch
  • Failed to get credentials: ... storage credential ...
  • PERMISSION_DENIED on external location validation.
• Several workspaces attached to the same metastore are affected at once.
• Started after an Azure cleanup, a resource group move, an RBAC change on the storage account, or a storage firewall change.

#Step 1: which credential is failing? (5 minutes)

SHOW STORAGE CREDENTIALS;
DESCRIBE STORAGE CREDENTIAL `sc-sales-prod`;
SHOW EXTERNAL LOCATIONS;
DESCRIBE EXTERNAL LOCATION `el-sales-prod`;

The credential shows the access connector resource ID (and managed identity ID for user-assigned). In the Catalog Explorer, External locations > Test connection reports which check fails (read, write, list, delete).

The metastore's managed storage uses its own credential as well; check Catalog > Metastore details if all managed tables fail.

#Step 2: does the access connector still exist?

az resource show --ids "/subscriptions/<sub>/resourceGroups/rg-dbx/providers/Microsoft.Databricks/accessConnectors/dbac-sales-prod" \
  --query "{name:name, identity:identity}" 2>&1 | head -20

Check the Activity Log for deletes on the connector or role assignments:

az monitor activity-log list --resource-group rg-dbx --offset 3d \
  --query "[?contains(operationName.value, 'accessConnectors') || contains(operationName.value, 'roleAssignments')].{t:eventTimestamp, op:operationName.value, who:caller, status:status.value}" -o table

#Step 3a: connector deleted, recreate it

Access connectors cannot be undeleted. Create a new one (a user-assigned identity is easier to keep stable in future):

az databricks access-connector create -g rg-dbx -n dbac-sales-prod -l westeurope --identity-type SystemAssigned
PID=$(az databricks access-connector show -g rg-dbx -n dbac-sales-prod --query identity.principalId -o tsv)

Grant it storage access (step 4), then point the storage credential at the new connector:

Run this as the credential owner or a metastore admin:

databricks storage-credentials update sc-sales-prod --json '{
  "azure_managed_identity": {
    "access_connector_id": "/subscriptions/<sub>/resourceGroups/rg-dbx/providers/Microsoft.Databricks/accessConnectors/dbac-sales-prod"
  }
}'

The name and ID of the storage credential stay the same, so external locations and tables that reference it keep working once the credential points at a valid identity.

Warning

Updating a storage credential affects every external location using it. Make sure the new identity has access to all of those storage paths before you switch.

#Step 3b: identity re-created with a new principal

Role assignments belong to the old principal ID and are orphaned ("Identity not found" in IAM). Re-grant to the new principal (step 4). The credential itself does not need to change if the connector resource ID is unchanged.

#Step 4: restore role assignments

SA=$(az storage account show -n stsalesprod -g rg-data --query id -o tsv)
az role assignment create --assignee-object-id $PID --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" --scope $SA
# file events / Auto Loader file notification mode may also need queue and event subscription roles
az role assignment list --scope $SA --query "[].{who:principalId, role:roleDefinitionName}" -o table

RBAC propagation takes a few minutes. Test again from the external location page.

#Step 5: storage networking

If identity and roles are fine, the storage firewall may now block Databricks:

az storage account show -n stsalesprod --query "networkRuleSet.{default:defaultAction, resourceAccessRules:resourceAccessRules, vnet:virtualNetworkRules[].virtualNetworkResourceId}"

For serverless compute and Unity Catalog access, the storage account needs a resource instance rule for the access connector (or trusted service access), plus network rules or private endpoints for classic compute subnets or serverless network connectivity configurations.

az storage account network-rule add -g rg-data --account-name stsalesprod \
  --resource-id "/subscriptions/<sub>/resourceGroups/rg-dbx/providers/Microsoft.Databricks/accessConnectors/dbac-sales-prod" \
  --tenant-id <tenant-id>

#Step 6: confirm

SELECT count(*) FROM sales_prod.silver.orders;
LIST 'abfss://lake@stsalesprod.dfs.core.windows.net/managed/';

• External location test connection passes all checks.
• Failed jobs repaired; streaming jobs restarted (they may have stopped on the errors).

#Prevent it next time

• Resource lock (CanNotDelete) on access connectors and on the storage accounts.
• User-assigned managed identities for connectors, managed in Terraform, so recreation keeps the principal stable.
• Exclude Databricks connectors from automated "unused resource" cleanups by tag.
• Alert on Activity Log deletes of Microsoft.Databricks/accessConnectors and role assignment deletes on lake storage.