Azure Databricks can't read ADLS: fixing 403 and PERMISSION_DENIED on Unity Catalog external locations

A query, Auto Loader stream or CREATE EXTERNAL TABLE fails against an abfss:// path. The error is either a Databricks PERMISSION_DENIED or an Azure 403 wrapped in an AbfsRestOperationException. There are five independent layers between a user and the bytes, and the error text usually tells you which one said no.

#Read the error first

Exact error class names vary by runtime version. Match on the Azure status text (AuthorizationPermissionMismatch vs AuthorizationFailure), because Azure Storage sets that part, not Databricks.

#Fast triage: test the credential directly

Run validation as a metastore admin or the credential owner. It tests Databricks → access connector → storage without involving the user's grants:

databricks external-locations get bronze_prod
databricks storage-credentials validate \
  --storage-credential-name lake_mi \
  --external-location-name bronze_prod

To test a path before an external location exists, pass --url abfss://bronze@stlakeprod.dfs.core.windows.net/ instead. In the UI, use Catalog Explorer → External data → External locations → (location) → Test connection.

• Validation fails: the problem is in Azure (role, firewall or connector). Work through sections 2 and 3.
• Validation passes but the user fails: the problem is Unity Catalog grants or compute. Work through sections 1, 4 and 5.

#1. Unity Catalog grants

SHOW EXTERNAL LOCATIONS;
DESCRIBE EXTERNAL LOCATION bronze_prod;
SHOW GRANTS ON EXTERNAL LOCATION bronze_prod;
SHOW GRANTS `alice@contoso.com` ON EXTERNAL LOCATION bronze_prod;

What each operation needs:

GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION bronze_prod TO `data-engineers`;
GRANT CREATE EXTERNAL TABLE ON EXTERNAL LOCATION bronze_prod TO `data-engineers`;

Two common gotchas:

• Reading an existing external table doesn't need any external location privilege. It needs SELECT on the table and USE on its catalog and schema. Don't hand out READ FILES to fix a table read, because that bypasses table-level controls such as row filters.
• Jobs run as the job owner or the configured Run as principal. Grant the privilege to that service principal, not to the person who reported the failure.

#2. The access connector's Azure role

Find the managed identity behind the credential, then check its roles on the storage account:

az databricks access-connector show -n ac-lake-prod -g rg-data-prod \
  --query "{type:identity.type, principalId:identity.principalId, uami:identity.userAssignedIdentities}" -o json

SA_ID=$(az storage account show -n stlakeprod -g rg-data-prod --query id -o tsv)

az role assignment list --assignee <principalId> --scope "$SA_ID" \
  --include-inherited -o table

You need Storage Blob Data Contributor at the account or container scope. Storage Blob Data Reader only works for credentials you deliberately mark read-only. Note that Owner or Contributor on the account are management-plane roles and do not grant data access.

az role assignment create \
  --assignee-object-id <principalId> \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" \
  --scope "$SA_ID"

Role assignments can take several minutes to apply. Wait, then re-run validate before you change anything else.

Also check:

• User-assigned identity: if the connector uses one, the storage credential must reference that identity's resource ID as well as the connector ID. Otherwise Databricks requests a token for the wrong identity.
• ADLS ACLs: if you grant access with POSIX ACLs instead of RBAC, the identity needs execute (--x) on every parent folder and read/write on the target. A missing execute bit on one parent folder also returns AuthorizationPermissionMismatch.
• Wrong connector: DESCRIBE STORAGE CREDENTIAL lake_mi shows which connector the credential points at. After a redeploy it's often an older one.

#3. Storage firewall and private endpoints

az storage account show -n stlakeprod -g rg-data-prod \
  --query "{public:publicNetworkAccess, default:networkRuleSet.defaultAction, bypass:networkRuleSet.bypass, instances:networkRuleSet.resourceAccessRules}" -o json

az network private-endpoint-connection list --id "$SA_ID" -o table

Add the resource instance rule:

az storage account network-rule add -g rg-data-prod --account-name stlakeprod \
  --resource-id "/subscriptions/<sub-id>/resourceGroups/rg-data-prod/providers/Microsoft.Databricks/accessConnectors/ac-lake-prod" \
  --tenant-id <tenant-id>

Check DNS from a notebook on the failing compute. A private endpoint that resolves to a public IP doesn't help:

import socket
print(socket.gethostbyname("stlakeprod.dfs.core.windows.net"))   # expect 10.x / private range

Tip

"Classic clusters work, SQL warehouse fails" almost always means the NCC private endpoint rule is missing or still Pending approval on the storage account.

#Confirm from the storage logs

If diagnostic settings send StorageRead/StorageWrite to Log Analytics:

StorageBlobLogs
| where TimeGenerated > ago(1h)
| where AccountName == "stlakeprod" and StatusCode == 403
| summarize count() by StatusText, AuthenticationType, CallerIpAddress, RequesterObjectId
| order by count_ desc

AuthorizationFailure with a public caller IP means a network problem. AuthorizationPermissionMismatch with the connector's object ID means a role problem.

#4. External location coverage

Unity Catalog decides which credential to use by matching the path against external location URLs.

• A path outside every external location fails, even if the identity has storage rights. Create a location for it, or use a broader parent location.
• External locations can't overlap each other or a catalog's managed storage. Create one location at the container or top-level folder and grant on that, rather than one per subfolder.
• Path-based access to a managed table's directory is blocked by design. Query the table by name.

CREATE EXTERNAL LOCATION landing_prod
  URL 'abfss://landing@stlakeprod.dfs.core.windows.net/'
  WITH (STORAGE CREDENTIAL lake_mi);

#5. Compute access mode

Unity Catalog credentials are only used on compute in Standard (formerly shared) or Dedicated (formerly single user) access mode, on SQL warehouses and on serverless. Access mode names vary by UI version. A cluster with No isolation shared access mode, or one with old fs.azure.account.key... / OAuth Spark configs, takes a different path and fails or uses legacy credentials.

databricks clusters get <cluster-id> | jq '{data_security_mode, spark_version, spark_conf}'

Expect data_security_mode to be USER_ISOLATION or SINGLE_USER (newer values may appear as DATA_SECURITY_MODE_STANDARD / DATA_SECURITY_MODE_DEDICATED). Remove legacy fs.azure.* keys from the Spark config.

#Prevent it next time

• Manage storage credentials, external locations, role assignments and NCC rules together in one Terraform module, so they can't drift apart.
• Grant READ FILES/WRITE FILES to groups and job service principals only. Give consumers table-level grants.
• Add databricks storage-credentials validate to your post-deployment checks for every environment.
• Send StorageBlobLogs to Log Analytics and alert on bursts of 403.
• Use cluster policies to enforce Unity Catalog access modes, so no-isolation clusters can't be created.