Connect Azure Databricks to ADLS with Unity Catalog: your first storage credential and external location

#The goal

Your data sits in an Azure Data Lake Storage (ADLS) account, and you want to read it from Databricks. You don't need storage keys or secrets in a notebook. Unity Catalog can connect using a managed identity, and it controls who can use that connection.

It has four parts, and each builds on the one before it:

Access connector (Azure identity)
   ↓ has a storage role on
ADLS account
   ↑ reached through
Storage credential (Unity Catalog object for the identity)
   ↓ used by
External location (credential + path)
   ↓ used by
Your tables, volumes and queries

#What you need

• An ADLS Gen2 account, with hierarchical namespace on.
• Rights to assign roles on that storage account.
• CREATE STORAGE CREDENTIAL and CREATE EXTERNAL LOCATION on the metastore. Metastore admins have these by default.

If you're new to Unity Catalog, read the basics first.

#Step 1: create an access connector

This Azure resource holds the managed identity:

RG=rg-data
LOC=centralindia

az databricks access-connector create \
  -g "$RG" -n ac-databricks-lake -l "$LOC" \
  --identity-type SystemAssigned

CONNECTOR_ID=$(az databricks access-connector show -g "$RG" -n ac-databricks-lake --query id -o tsv)
PRINCIPAL=$(az databricks access-connector show -g "$RG" -n ac-databricks-lake --query identity.principalId -o tsv)
echo "$CONNECTOR_ID"

You may be asked to install the databricks CLI extension on first use. You can also create it from the Azure portal by searching for Access Connector for Azure Databricks.

#Step 2: grant it access to the storage

STORAGE_ID=$(az storage account show -g "$RG" -n stlake01 --query id -o tsv)

az role assignment create \
  --assignee-object-id "$PRINCIPAL" \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" \
  --scope "$STORAGE_ID"

Use Storage Blob Data Reader if you'll only ever read. You can also scope the role to one container, rather than the whole account. Role assignments take a few minutes to apply.

Tip

Features such as Auto Loader's file notifications may need extra roles. You can add those later, as you need them.

#Step 3: register the storage credential

databricks storage-credentials create --json '{
  "name": "cred_lake",
  "comment": "Access connector for stlake01",
  "azure_managed_identity": {
    "access_connector_id": "'"$CONNECTOR_ID"'"
  }
}'

You can also do this in Catalog Explorer, under External data, then Credentials.

#Step 4: create the external location

Run this in a notebook or the SQL editor. Use the abfss format, with the DFS endpoint:

CREATE EXTERNAL LOCATION IF NOT EXISTS raw_lake
URL 'abfss://raw@stlake01.dfs.core.windows.net/'
WITH (STORAGE CREDENTIAL cred_lake)
COMMENT 'Raw zone of stlake01';

Then test it:

databricks storage-credentials validate --json '{
  "storage_credential_name": "cred_lake",
  "url": "abfss://raw@stlake01.dfs.core.windows.net/"
}'

Catalog Explorer also has a Test connection button on the location's page. Look at the results for read, list and write.

#Step 5: grant access to people

An external location doesn't give anyone access until you grant it. Use a group:

GRANT READ FILES ON EXTERNAL LOCATION raw_lake TO `data-engineers`;
GRANT WRITE FILES ON EXTERNAL LOCATION raw_lake TO `data-engineers`;
GRANT CREATE EXTERNAL TABLE ON EXTERNAL LOCATION raw_lake TO `data-engineers`;
GRANT CREATE EXTERNAL VOLUME ON EXTERNAL LOCATION raw_lake TO `data-engineers`;

See the grants cheat sheet for the full list. Check who has what:

SHOW GRANTS ON EXTERNAL LOCATION raw_lake;

#Step 6: read the data

List the files, using compute in Standard or Dedicated access mode:

LIST 'abfss://raw@stlake01.dfs.core.windows.net/orders/';

Read a CSV folder directly:

SELECT *
FROM read_files(
  'abfss://raw@stlake01.dfs.core.windows.net/orders/',
  format => 'csv',
  header => true
)
LIMIT 10;

#Step 7: make it easier to use

Long abfss paths are awkward, so give the data a proper name.

Use an external volume for files:

CREATE EXTERNAL VOLUME main.raw.orders_files
LOCATION 'abfss://raw@stlake01.dfs.core.windows.net/orders/';

GRANT READ VOLUME ON VOLUME main.raw.orders_files TO `data-analysts`;

Then the path is simply /Volumes/main/raw/orders_files/. Analysts can read those files without any rights on the external location itself. See volumes and loading a CSV from a volume.

Use an external table for Delta data:

CREATE TABLE main.raw.orders
LOCATION 'abfss://raw@stlake01.dfs.core.windows.net/delta/orders/';

If you don't have a reason to keep your own paths, consider a managed table instead. See managed and external tables.

#Common errors

For a full walk-through of 403 errors, read external location 403.

#Check for storage networking

If the storage account has a firewall, the steps above won't be enough. Check it:

az storage account show -g "$RG" -n stlake01 \
  --query "{public:publicNetworkAccess, default:networkRuleSet.defaultAction, instanceRules:networkRuleSet.resourceAccessRules}"

If default is Deny, add a resource instance rule for the access connector, as well as a network path for your compute. Serverless needs network connectivity configuration.

Warning

Don't delete an access connector that's in use. Every table and location that relies on it will start failing with 403 errors. See access connector deleted for what that looks like.

#Prevent it next time

• Use one connector per environment, and create it with infrastructure as code.
• Grant access to groups, never to individuals.
• Keep external locations at a narrow path, such as a single container or zone.
• Don't write directly to these paths from outside Unity Catalog. It makes access and lineage harder to track.