The storage layer under Azure Databricks: choosing ADLS Gen2 settings and wiring secure, keyless access

Your lakehouse lives in Azure Storage, not inside Databricks. The account type, performance tier, redundancy and network settings you choose decide cost, speed, DR options and security. Changing them later means moving data.

#The options

Warning

Hierarchical namespace must be chosen when the account is created. There's an in-place upgrade path, but it has conditions. Plan for HNS from the start.

#Performance tier and access tier

• Standard (hot) for active lake data. This is the default choice.
• Premium block blob where many small reads and writes dominate.
• Cool/Cold/Archive only for data that is not part of a live Delta table, such as raw file archives. Tiering files under a Delta table breaks reads (archive) or makes them slow and expensive (cool/cold).

#Redundancy and DR

For a tested DR, most teams use DEEP CLONE or job-driven replication to a second-region account, rather than raw GRS failover. See the Databricks DR post.

#Layout

• One storage account per environment at minimum (dev, test, prod). Separate accounts per data domain if you need different owners, networks or keys.
• Containers per purpose: landing, bronze, silver, gold, and one per Unity Catalog managed location.
• Don't use the workspace's own managed resource group storage (DBFS root) for business data.
• Stay well below the per-account request and throughput limits. Very busy platforms spread load over several accounts.

#Connecting securely: Unity Catalog

The recommended pattern needs no keys anywhere:

1. Create an Access Connector for Azure Databricks (a resource with a managed identity).
2. Grant the connector's identity Storage Blob Data Contributor on the storage account or container.
3. In Databricks, create a storage credential that uses the access connector.
4. Create external locations for the paths, then grant users rights on them.

CREATE STORAGE CREDENTIAL lake_mi
  WITH (AZURE_MANAGED_IDENTITY = (ACCESS_CONNECTOR_ID =
    '/subscriptions/0000-0000/resourceGroups/rg-data-prod/providers/Microsoft.Databricks/accessConnectors/ac-lake-prod'));

CREATE EXTERNAL LOCATION bronze_prod
  URL 'abfss://bronze@stlakeprod.dfs.core.windows.net/'
  WITH (STORAGE CREDENTIAL lake_mi);

GRANT READ FILES, WRITE FILES, CREATE EXTERNAL TABLE ON EXTERNAL LOCATION bronze_prod TO `data-engineers`;

The CREATE STORAGE CREDENTIAL SQL form isn't supported in every runtime. If it isn't in yours, create the credential in Catalog Explorer → External data → Credentials, or with Terraform, which is the usual choice for production.

#Legacy access methods (avoid for new work)

#Network security

• Set the storage firewall to deny public access.
• Create private endpoints for both the dfs and blob sub-resources in a VNet that your Databricks compute can reach. Link privatelink.dfs.core.windows.net and privatelink.blob.core.windows.net private DNS zones.
• Classic compute in an injected VNet can alternatively use service endpoints on the Databricks subnets.
• Serverless compute: configure a network connectivity configuration (NCC) in the account console. Add private endpoint rules to the storage account, then approve them on the storage side.
• Allow the access connector as a resource instance exception if you use the storage firewall without private endpoints.

#Data protection settings

#Checklist

• StorageV2 with hierarchical namespace; ZRS (or better) for prod.
• Separate accounts per environment; containers per layer and managed location.
• Access connector + storage credential + external locations; no keys, SAS or mounts.
• Public access disabled; private endpoints (dfs + blob) with DNS; NCC for serverless.
• Soft delete on; no lifecycle rules on Delta table paths.
• Monitoring for throttling and capacity.