ADLS firewall blocking Azure Databricks: 403 AuthorizationFailure from classic clusters, Unity Catalog and serverless - resource instance rules, subnets and NCCs

#Symptoms

• Queries fail with 403 This request is not authorized to perform this operation / AuthorizationFailure right after the storage account's Public network access was set to "Enabled from selected networks" or "Disabled".
• It fails on some compute but not others: classic clusters work, SQL serverless doesn't; or the reverse.
• Validating a storage credential or external location in Catalog Explorer fails with a network error, although RBAC is correct.

AuthorizationFailure from storage with correct RBAC is almost always the storage firewall. (Wrong RBAC usually shows AuthorizationPermissionMismatch.)

#Three different callers

Azure Databricks reaches storage from different places, and each needs its own allowance:

Fixing only one of them gives "works here, fails there".

#Step 1: check the storage firewall

az storage account show -n stlakeprod -g rg-data \
  --query "{public:publicNetworkAccess, default:networkRuleSet.defaultAction, bypass:networkRuleSet.bypass, vnets:networkRuleSet.virtualNetworkRules[].virtualNetworkResourceId, instances:networkRuleSet.resourceAccessRules}"

#Step 2: allow classic compute

Service endpoint route (simple, no extra cost):

for s in snet-dbx-host snet-dbx-container; do
  az network vnet subnet update -g rg-dbx-net --vnet-name vnet-dbx -n $s --service-endpoints Microsoft.Storage
  SUBNET_ID=$(az network vnet subnet show -g rg-dbx-net --vnet-name vnet-dbx -n $s --query id -o tsv)
  az storage account network-rule add -g rg-data --account-name stlakeprod --subnet $SUBNET_ID
done

Or private endpoints for dfs and blob in the Databricks VNet (or a peered hub), with private DNS zones privatelink.dfs.core.windows.net and privatelink.blob.core.windows.net resolvable from the cluster subnets. Private endpoints are required if public access is fully disabled.

#Step 3: allow the Unity Catalog access connector

Add a resource instance rule for the access connector:

CONNECTOR_ID=$(az databricks access-connector show -g rg-dbx -n ac-dbx-prod --query id -o tsv)
TENANT=$(az account show --query tenantId -o tsv)

az storage account network-rule add -g rg-data --account-name stlakeprod \
  --resource-id $CONNECTOR_ID --tenant-id $TENANT

(The az databricks access-connector commands come from the databricks CLI extension; you can also copy the connector's resource ID from the portal.)

Then validate the storage credential/external location in Catalog Explorer (Test connection).

Tip

The resource instance rule only covers requests made as the access connector's identity from Azure Databricks. It doesn't open the account to anything else, so it's preferable to enabling broad "trusted Microsoft services" bypass.

#Step 4: allow serverless compute

1. Create an NCC in the account console (or account API) in the same region as the workspace and attach it to the workspace.
2. Either:
   • Stable subnets: copy the NCC's subnet IDs and add them as VNet rules on the storage firewall, or
   • Private endpoint rules: add a private endpoint rule in the NCC for the storage account (dfs and blob), then approve the pending private endpoint connection on the storage account.

az storage account show -n stlakeprod -g rg-data \
  --query "privateEndpointConnections[].{name:name,status:privateLinkServiceConnectionState.status}" -o table

az storage account private-endpoint-connection approve -g rg-data --account-name stlakeprod \
  --name <connection-name> --description "Databricks serverless NCC"

NCC changes can take some minutes to apply, and serverless resources may need to restart to pick them up.

#Step 5: test each path

-- On a classic cluster
LIST 'abfss://curated@stlakeprod.dfs.core.windows.net/sales/';

-- On a serverless SQL warehouse
SELECT COUNT(*) FROM main.sales.orders;

And check storage logs for the denied caller:

StorageBlobLogs
| where TimeGenerated > ago(1h) and AccountName == "stlakeprod" and StatusCode == 403
| summarize count() by StatusText, CallerIpAddress, AuthenticationType, bin(TimeGenerated, 5m)

CallerIpAddress tells you whether a request came from your VNet (private/service endpoint), from serverless, or from somewhere unexpected.

#Decision table

#Prevent it next time

• Design storage network access for all three callers before turning on the firewall.
• Keep firewall rules, resource instance rules and NCC private endpoints in Terraform.
• After any storage network change, run a test query from a classic cluster and a serverless warehouse.
• Keep storage resource logs on for lake accounts so 403s can be traced quickly.