Sev1: Unity Catalog tables fail with 403 after a storage firewall change — network rules for serverless and classic compute

#Symptoms

• This request is not authorized to perform this operation / AuthorizationFailure (403) reading or writing tables.
• Works from some compute (classic clusters) but fails from serverless SQL warehouses, or the reverse.
• The access connector and its role assignment are still in place.

#1. Confirm it is the network, not RBAC

az storage account show -g rg-lake -n stlakeprod \
  --query "{public:publicNetworkAccess, default:networkRuleSet.defaultAction, vnets:networkRuleSet.virtualNetworkRules[].id, ips:networkRuleSet.ipRules[].ipAddressOrRange, resourceAccess:networkRuleSet.resourceAccessRules}"

Then check the storage activity log for the change:

az monitor activity-log list -g rg-lake --offset 24h \
  --query "[?contains(operationName.value,'storageAccounts/write')].{t:eventTimestamp, who:caller}" -o table

The 403 from storage with AuthorizationFailure (not AuthorizationPermissionMismatch) usually means a network rule blocked the request.

#2. Classic compute (VNet-injected clusters)

Classic clusters reach storage from your VNet. Options:

• Private endpoint for the dfs (and blob) sub-resource in a subnet the workspace VNet can reach, with the privatelink.dfs.core.windows.net zone linked.
• Service endpoint: add the workspace's host and container subnets to the storage account's VNet rules.

az storage account network-rule add -g rg-lake --account-name stlakeprod \
  --subnet /subscriptions/<sub>/resourceGroups/rg-dbx/providers/Microsoft.Network/virtualNetworks/vnet-dbx/subnets/snet-dbx-container

#3. Serverless compute

Serverless runs in the Databricks-managed network, not your VNet. With storage firewall on, you need one of:

• A network connectivity configuration (NCC) in the Databricks account, attached to the workspace, with either stable service-endpoint subnets added to the storage firewall, or private endpoint rules to the storage account (which you then approve on the storage side).
• Approve pending private endpoint connections:

az network private-endpoint-connection list --id <storage-account-id> \
  --query "[].{name:name, status:properties.privateLinkServiceConnectionState.status}" -o table
az network private-endpoint-connection approve --id <connection-id> --description "Databricks serverless NCC"

Check the current Azure Databricks serverless networking documentation; options and names change as the feature evolves.

#4. Trusted access for the access connector

Allow the Databricks access connector as a resource instance so Unity Catalog's storage credential calls are permitted through the firewall:

az storage account network-rule add -g rg-lake --account-name stlakeprod \
  --resource-id /subscriptions/<sub>/resourceGroups/rg-dbx/providers/Microsoft.Databricks/accessConnectors/ac-unity \
  --tenant-id <tenant-id>

Warning

Do not "fix" this by setting defaultAction=Allow on a production data lake. If you must as a last resort, record it, restrict by IP where possible, and close it within hours.

#5. Verify from both compute types

-- On a classic cluster and on a serverless SQL warehouse
SELECT COUNT(*) FROM prod.sales.orders;
LIST 'abfss://lake@stlakeprod.dfs.core.windows.net/sales/orders/';

The LIST uses the external location's credential and confirms the path end to end.

#6. External locations

Validate each external location in Catalog Explorer (Test connection) or:

DESCRIBE EXTERNAL LOCATION ext_sales;

#Prevent it next time

• Treat storage network rules as part of the Databricks platform; change them only with the data platform team.
• Document which subnets, NCCs and resource instances must stay allowed.
• Put a resource lock and IaC ownership on lake storage accounts.
• Add a synthetic query from serverless and classic compute that runs every 15 minutes.