Introduction to Unity Catalog volumes on Azure Databricks: governed storage for files that aren't tables

#Why volumes exist

Tables cover structured data. But pipelines also deal with files that aren't tables: landing CSVs, JSON from APIs, images, PDFs for RAG, ML model artefacts, wheel files, init scripts. In the past people used DBFS mounts or direct abfss:// paths with keys, which bypassed governance.

A volume is a Unity Catalog object for files. It lives in a schema, next to tables:

catalog.schema.table      -> structured, queried with SQL
catalog.schema.volume     -> files, accessed as /Volumes/catalog/schema/volume/...

Access is controlled with the same GRANT model and is recorded in audit logs.

#Managed vs external volumes

#Step 1: Create a volume

-- Managed
CREATE VOLUME IF NOT EXISTS main.sales.uploads
COMMENT 'Ad hoc files uploaded by the sales analytics team';

-- External: needs an external location covering the path
CREATE EXTERNAL VOLUME IF NOT EXISTS main.sales.landing
LOCATION 'abfss://landing@stdatalakeprod.dfs.core.windows.net/sales/'
COMMENT 'Files dropped by ADF from the ERP export';

For the external volume you need CREATE EXTERNAL VOLUME on the external location, plus USE CATALOG, USE SCHEMA and CREATE VOLUME on the schema. The external location is backed by a storage credential, usually an Azure Databricks access connector (managed identity) with Storage Blob Data Contributor on the storage account or container.

SHOW VOLUMES IN main.sales;
DESCRIBE VOLUME main.sales.landing;

#Step 2: Grant access

GRANT USE CATALOG ON CATALOG main TO `data-engineers`;
GRANT USE SCHEMA  ON SCHEMA  main.sales TO `data-engineers`;

GRANT READ VOLUME, WRITE VOLUME ON VOLUME main.sales.landing TO `data-engineers`;
GRANT READ VOLUME               ON VOLUME main.sales.landing TO `sales-analysts`;

Volume privileges are just READ VOLUME and WRITE VOLUME (plus ownership). You can't grant access to a sub-folder; create separate volumes if teams need separate areas.

#Step 3: Work with files

#Python / PySpark

base = "/Volumes/main/sales/landing"

# List
display(dbutils.fs.ls(f"{base}/orders/2026/11/01/"))

# Read with Spark
df = (spark.read.option("header", True)
        .csv(f"{base}/orders/2026/11/01/"))

# Plain Python file APIs also work on /Volumes paths
import os, json
with open(f"{base}/config/mapping.json") as f:
    mapping = json.load(f)
os.makedirs(f"{base}/exports/2026-11-02", exist_ok=True)

#SQL

SELECT * FROM read_files(
  '/Volumes/main/sales/landing/orders/2026/11/01/',
  format => 'csv', header => true);

LIST '/Volumes/main/sales/landing/orders/';

#Auto Loader from a volume

(spark.readStream.format("cloudFiles")
   .option("cloudFiles.format", "json")
   .option("cloudFiles.schemaLocation", "/Volumes/main/sales/checkpoints/orders_schema")
   .load("/Volumes/main/sales/landing/orders/")
   .writeStream
   .option("checkpointLocation", "/Volumes/main/sales/checkpoints/orders")
   .trigger(availableNow=True)
   .toTable("main.sales.orders_bronze"))

#Databricks CLI (from your laptop or a pipeline)

databricks fs ls dbfs:/Volumes/main/sales/uploads/
databricks fs cp ./price_list.csv dbfs:/Volumes/main/sales/uploads/price_list.csv

The CLI uses the dbfs:/Volumes/... form for volume paths.

#Things that need a supported compute mode

Volumes need Unity Catalog-enabled compute: clusters in standard (shared) or dedicated (single user) access mode, SQL warehouses or serverless. Older "no isolation shared" clusters can't use them.

Tip

Cluster libraries and init scripts can be stored in volumes too (/Volumes/main/platform/libs/mylib-1.2.0-py3-none-any.whl). On standard access mode, an admin must add the path to the metastore allowlist first.

#Common errors

#Migrating from DBFS mounts

Replace /mnt/landing/... with /Volumes/<catalog>/<schema>/<volume>/..., one mount at a time:

1. Create an external location and external volume for the same storage path.
2. Grant volume access to the groups that used the mount.
3. Search code for /mnt/ and dbfs:/mnt/, replace paths, test.
4. Unmount (dbutils.fs.unmount("/mnt/landing")) once nothing uses it.

#Prevent problems later

• One volume per purpose and owner (landing, exports, checkpoints, libs), not one big bucket-of-everything.
• Use external volumes for anything other systems write to; managed volumes for Databricks-only files.
• Keep checkpoints in a dedicated volume that only job service principals can write.
• Review volume grants quarterly with SHOW GRANTS ON VOLUME ....