Fixing file-not-found for good: a map of the Azure Databricks file systems and when to use each

"File not found" is one of the most common Databricks errors. The cause is usually a path that is valid in one file system but not the one being used. There are four places files can live, and each has its own path style.

#The four file systems

Note

Databricks recommends volumes for files and tables for data. DBFS mounts and the DBFS root are deprecated for data storage. Shared-access compute and serverless restrict them.

#Which API reads which path

Key trap: in dbutils.fs and Spark, a bare path like /tmp/x means DBFS. In Python and %sh, it means the driver's local disk. Volumes are the exception: /Volumes/... means the same thing everywhere.

#Common operations

# list, copy, move, delete
dbutils.fs.ls("/Volumes/main/raw/landing/")
dbutils.fs.cp("/Volumes/main/raw/landing/a.csv", "/Volumes/main/raw/archive/a.csv")
dbutils.fs.mv("/Volumes/main/raw/landing/b.csv", "/Volumes/main/raw/archive/b.csv")
dbutils.fs.rm("/Volumes/main/raw/tmp/", recurse=True)
dbutils.fs.mkdirs("/Volumes/main/raw/landing/2026/10/23")
print(dbutils.fs.head("/Volumes/main/raw/landing/a.csv", 500))

# local driver disk to volume (e.g. after downloading a file in Python)
import urllib.request
urllib.request.urlretrieve("https://example.com/data.csv", "/tmp/data.csv")
dbutils.fs.cp("file:/tmp/data.csv", "/Volumes/main/raw/landing/data.csv")

# volumes work with plain Python too
import pandas as pd
pdf = pd.read_csv("/Volumes/main/raw/landing/data.csv")

%sh
ls -lh /Volumes/main/raw/landing/
df -h /local_disk0          # driver scratch space
du -sh /tmp/* 2>/dev/null | sort -h | tail

#Creating and governing volumes

-- managed volume: storage handled by Unity Catalog
CREATE VOLUME main.raw.landing COMMENT 'Inbound files from partners';

-- external volume over an existing path (needs an external location)
CREATE EXTERNAL VOLUME main.raw.partner_drop
LOCATION 'abfss://partner@stlakeprod.dfs.core.windows.net/drop';

GRANT READ VOLUME, WRITE VOLUME ON VOLUME main.raw.landing TO `data-engineers`;
GRANT READ VOLUME ON VOLUME main.raw.landing TO `data-analysts`;
SHOW VOLUMES IN main.raw;

-- query files in a volume straight from SQL
SELECT * FROM read_files('/Volumes/main/raw/landing/', format => 'csv', header => true) LIMIT 10;
LIST '/Volumes/main/raw/landing/';

#From your laptop: Databricks CLI

databricks auth login --host https://adb-1234567890123456.7.azuredatabricks.net
databricks fs ls dbfs:/Volumes/main/raw/landing
databricks fs cp ./sales.csv dbfs:/Volumes/main/raw/landing/sales.csv
databricks fs cp -r ./batch/ dbfs:/Volumes/main/raw/landing/batch/ --overwrite
databricks workspace import-dir ./notebooks /Workspace/Users/me@corp.com/project

The CLI addresses volumes with the dbfs:/Volumes/... form. That's a CLI convention; the files are still in the volume, not the DBFS root.

You can also upload files through the UI. In Catalog Explorer, open the volume and choose Upload to this volume.

#Moving off DBFS and mounts

1. Inventory: display(dbutils.fs.mounts()) and dbutils.fs.ls("dbfs:/FileStore"). Search code for /mnt/, dbfs:/ and /dbfs/.
2. For each mount, create a storage credential + external location on the same storage path, then an external volume or external tables over it.
3. Replace /mnt/raw/... with /Volumes/main/raw/partner_drop/... in code.
4. Move small files in dbfs:/FileStore to a managed volume or workspace files.
5. Move init scripts and libraries from DBFS to volumes or workspace files, and allow-list them.
6. Unmount (dbutils.fs.unmount("/mnt/raw")) once nothing references it.

#Driver disk housekeeping

Long-running interactive clusters can fill the driver disk with temp files, logs or downloads. That causes No space left on device errors and driver restarts.

• Write large outputs to volumes, not /tmp.
• Clean up in the notebook: dbutils.fs.rm("file:/tmp/mywork", True).
• Use auto-termination, so restarted clusters start clean.
• Collecting a large result to the driver (toPandas() on big data) fills memory and disk. Write to a table or volume instead.
• If /local_disk0 fills on workers, check for shuffle spill. That's a sizing problem (see the job tuning post).

#Quick decision

• Data you query with SQL: table.
• Files (CSV drops, PDFs, images, model artefacts, checkpoints): volume.
• Code and small config: workspace files in a Git folder.
• Temporary scratch: driver local disk, and expect it to vanish.
• New data in DBFS root or mounts: no.