Azure Databricks \"No space left on device\": fixing full driver and worker disks

#Symptoms

One or more of these, usually after a job or cluster has been running for a while:

• java.io.IOException: No space left on device in a failed Spark task or the driver log.
• OSError: [Errno 28] No space left on device from Python code, pip install or a file write.
• Tasks fail with FetchFailedException or executors are lost after heavy shuffle.
• The driver restarts ("The Spark driver has stopped unexpectedly and is restarting") on a long-running all-purpose cluster.

The fix depends on which node filled up. Workers and the driver behave differently on Azure Databricks.

#How local disk works on Azure Databricks

• Each VM has local storage. Spark puts shuffle files, spill and the disk cache under /local_disk0.
• Autoscaling local storage is on for all Azure Databricks compute. When a worker runs low on disk, Databricks attaches another managed disk, up to 5 TB total per VM. The extra disks stay until the VM is released.
• That monitoring is for Spark workers. The driver doesn't get the same protection, so a full driver disk is the most common case on interactive clusters.
• Serverless compute doesn't expose disk settings. If you see this error on serverless, the fix is in the code (write to volumes, reduce spill), not in configuration.

#1. Find which node is full

#Driver

From a notebook attached to the cluster:

%sh
df -h / /tmp /local_disk0 2>/dev/null
du -xsh /tmp/* /local_disk0/* 2>/dev/null | sort -h | tail -15

If / or /tmp is at 100%, the driver is full. On standard (shared) access mode, %sh runs as a restricted user and may not see every directory; use a dedicated cluster or the cluster logs if you need the full picture.

#Workers

• Compute → your cluster → Metrics: look at the hardware metrics per node. Free filesystem space shows which nodes are running out. Metric names vary by release; check what your workspace shows.
• Spark UI → Stages: sort the failed stage by Shuffle Spill (Disk) and Shuffle Write. One task with far more spill than the others means skew.
• Cluster event log: check whether autoscaling local storage expanded disks or failed to.

databricks clusters events -p prod --json '{
  "cluster_id": "0123-456789-abcde123",
  "event_types": ["EXPANDED_DISK", "DID_NOT_EXPAND_DISK"],
  "limit": 50
}'

DID_NOT_EXPAND_DISK with a message in the details means Databricks tried to attach disk and couldn't. Read the reason before changing anything.

#2. Causes and fixes

#Driver: stop writing to local disk

The usual pattern is code built for a laptop:

# Fills the driver disk with a large file
pdf = spark.table("sales.gold.orders").toPandas()
pdf.to_csv("/tmp/orders_export.csv")

Write with Spark straight to a volume, or at least put the file in one:

(spark.table("sales.gold.orders")
      .coalesce(1)
      .write.mode("overwrite")
      .option("header", True)
      .csv("/Volumes/sales/exports/files/orders_export"))

Clean up a full driver straight away, so you can keep working while you fix the code:

%sh
# Look first, then delete only files you or your job created
ls -lah /tmp | sort -k5 -h | tail -20
rm -rf /tmp/orders_export.csv /tmp/my_job_scratch

Caution

Don't rm -rf /local_disk0/* or delete Spark directories under /tmp on a running cluster. Running tasks, shuffle files and the disk cache live there. Removing them fails queries for every user on the cluster. If you can't tell what a file is, restart the cluster instead.

If the driver really needs more local disk (for example, single-node ML that downloads large models), pick a driver type with a bigger local temp disk. On Azure, VM sizes with a d in the name (such as the Ddsv5 and Edsv5 series) include local temp storage; check the size of each SKU before you choose.

#Workers: fix the skew instead of adding disk

Check for a skewed key:

from pyspark.sql import functions as F

(spark.table("sales.silver.events")
      .groupBy("customer_id")
      .count()
      .orderBy(F.desc("count"))
      .show(20))

If a few keys hold most of the rows:

• Make sure AQE and its skew join handling are on (they are by default on current runtimes):

  spark.conf.get("spark.sql.adaptive.enabled")
  spark.conf.get("spark.sql.adaptive.skewJoin.enabled")
  

• Filter out nulls or placeholder keys (-1, 'UNKNOWN') before joins. These are often the hot key.

• Broadcast the small side of the join so the big side doesn't shuffle:

  big.join(F.broadcast(small_dim), "customer_id")
  

• Salt very hot keys if AQE doesn't split them enough.

More partitions also spread spill over more tasks and nodes, but they don't help when one key holds most of the data.

#Workers: cap the disk cache

The disk cache keeps copies of remote Parquet/Delta data on local SSD. On cache-heavy clusters it can take a large share of the disk. Cap it in the cluster's Spark config:

spark.databricks.io.cache.maxDiskUsage 50g

Or turn it off for a cluster that mostly runs large one-pass ETL:

spark.databricks.io.cache.enabled false

Test before changing it on clusters that serve repeated interactive queries. Without the cache, those queries read from ADLS every time.

#3. Verify the fix

1. Rerun the failed job or repair only the failed tasks:

   databricks jobs repair-run <run-id> --rerun-all-failed-tasks -p prod
   

2. Watch Shuffle Spill (Disk) in the Spark UI for the same stage. It should be lower and spread evenly across tasks.

3. Check df -h on the driver at the end of the run.

4. Check the cluster event log for EXPANDED_DISK. A few are normal. Many on every run means the job is still spilling more than it should.

#Prevent it next time

• Never write job output to local paths. Review code for /tmp, file:/ and toPandas() on large tables; send files to Unity Catalog volumes.

• Set auto-termination on all-purpose clusters with a compute policy, so drivers don't build up weeks of temp files and logs.

• Use job compute for production. Each run gets fresh VMs and a clean disk.

• Alert on duration: a job that suddenly spills much more data usually also runs longer. A duration threshold on the job catches it before the disk fills.

• Turn on cluster log delivery to a volume, so you still have the driver and executor logs after the cluster terminates:

  "cluster_log_conf": { "volumes": { "destination": "/Volumes/ops/logs/cluster-logs" } }
  

• Track tables whose data grows fast and recheck partitioning and join strategy when they double in size.