Home / Databricks on Azure

Databricks on AzureTroubleshootingIntermediate

Databricks \"driver is temporarily unavailable\" or \"not responding\" - fixing driver overload, GC pauses and collect()

The Spark driver stops responding when it runs out of memory or spends its time in garbage collection, usually from collect(), toPandas(), large broadcasts or too many notebooks on one cluster. Find which one, then fix the code or resize the driver.

Deepak Bhardwaj5 min read

Applies toAzure Databricks all-purpose and job compute (Databricks Runtime 13.x LTS and later); PySpark and Scala notebooks and jobs

Symptoms

  • A notebook shows "The spark driver has stopped unexpectedly and is restarting" or "Driver is temporarily unavailable".
  • A job fails with "Driver is not responding, likely due to GC" or the run is "Lost" partway through.
  • Cells hang for minutes with no Spark jobs running.

The driver is a single JVM (plus a Python process for PySpark). It plans queries, tracks tasks, holds broadcast variables, and receives every result you bring back to the notebook. When it runs short of memory, it spends its time in garbage collection and stops answering heartbeats.

1. Find the cause in the driver logs

Open Compute > your cluster > Driver logs (or the job run's Spark UI > Driver logs):

Log pattern Likely cause
java.lang.OutOfMemoryError: Java heap space Driver JVM out of memory
Long Full GC or GC overhead limit exceeded Heap nearly full, GC thrashing
Total size of serialized results ... is bigger than spark.driver.maxResultSize Too much data returned to the driver
Python process killed, Fatal Python error Python side out of memory (pandas, toPandas)
Many notebooks attached, steady growth Shared all-purpose cluster overloaded

Also look at the Metrics tab of the cluster for driver memory and CPU over time.

2. Common code causes

Code Why it hurts Instead
df.collect() / df.toPandas() on large data Every row goes to the driver Write results to a table; display() a sample; limit() first
df.count() in a loop Many small jobs, plan rebuilding Count once, or use aggregates
Python loop building a huge DataFrame union Query plan grows huge on the driver Build the list and unionByName once, or use a single write per batch
broadcast() on a large table Driver collects then sends it Broadcast only small dimension tables; check spark.sql.autoBroadcastJoinThreshold
Reading thousands of small files with listing on the driver File index held in driver memory Compact files; use Auto Loader for incremental ingestion
Large pandas work in a notebook Runs only on the driver Use pandas API on Spark or Spark DataFrames

Find which cell did it: the cell that was running at the time of the restart is in the notebook's revision history and the job's task output.

# safe preview instead of toPandas() on the full dataset
display(df.limit(1000))

# if you really need pandas, check the size first
print(df.count(), len(df.columns))
pdf = df.limit(200_000).toPandas()

3. Long-running notebook state

Iterative notebooks keep creating cached DataFrames, broadcast variables and Python objects:

spark.catalog.clearCache()
import gc; gc.collect()

Detaching and re-attaching the notebook clears its Python state. Restarting the cluster clears everything.

4. Shared all-purpose clusters

Many users on one cluster share one driver. Each attached notebook adds a REPL and its state.

  • Move scheduled work to job compute or serverless jobs, so each job gets its own driver.
  • Give heavy interactive users their own cluster or a SQL warehouse.
  • Check the attached notebooks under Compute > cluster > Notebooks.

5. Resize the driver (after fixing the code)

  • Pick a larger or memory-optimised driver type separately from workers (for example an E-series driver with D-series workers).
  • Raise spark.driver.maxResultSize only if you deliberately return a larger result. It's a guard rail, not a fix.
"spark_conf": {
  "spark.driver.maxResultSize": "8g"
}

Warning

Raising maxResultSize without a larger driver just moves the failure from a clean error to an out-of-memory crash. Size the driver memory well above the result size.

6. Jobs: make the failure clear and recoverable

  • Set task retries for transient driver loss, but only for idempotent tasks.
  • Split huge notebooks into multiple job tasks. Each task gets a clean Python state on job compute.
  • Write intermediate results to Delta tables instead of keeping them in memory between steps.

Quick checks

spark.conf.get("spark.driver.maxResultSize")
spark.sparkContext.getConf().get("spark.driver.memory", "default")
spark.conf.get("spark.sql.autoBroadcastJoinThreshold")

The driver memory setting is fixed by the node type unless overridden. On Databricks you change it by choosing a different driver node type.

Prevent it next time

  • Code review rule: no collect()/toPandas() without a limit or a size check.
  • Production pipelines on job compute, not shared all-purpose clusters.
  • Alert on job runs that fail with driver errors, and look at the trend per job.

Checklist

  • Driver log read: heap OOM, GC, maxResultSize or Python OOM
  • Offending cell or task found
  • collect/toPandas/broadcast patterns fixed
  • Heavy work moved off shared clusters
  • Driver sized for the remaining legitimate need
  • Retries only on idempotent tasks

Related fixes