Home / Databricks on Azure

Databricks on AzureTroubleshootingAdvanced

Sev1: Databricks job fails with 'driver is up but is not responsive' — driver OOM and GC storms

A critical Azure Databricks job dies with driver unresponsive, driver restarted or OutOfMemoryError. Find what pulls data to the driver, fix the code or driver size, and get the run through.

Deepak Bhardwaj3 min read

Applies toAzure Databricks (Databricks Runtime 13.3 LTS+), PySpark, Databricks CLI

Symptoms

  • Run fails with The spark driver has stopped unexpectedly and is restarting, Driver is up but is not responsive, likely due to GC, or java.lang.OutOfMemoryError: Java heap space.
  • Notebook cells hang, then the cluster detaches.
  • The same job worked yesterday with less data.

1. Confirm it is the driver

  • Cluster Event log: look for DRIVER_NOT_RESPONDING, DRIVER_UNAVAILABLE or driver restarts.
  • Driver logs (stderr, log4j): search for OutOfMemoryError or long GC pauses.
  • Metrics tab: driver memory near 100% and CPU stuck in GC.
databricks clusters events <cluster-id> --output json | jq '.events[] | select(.type|test("DRIVER")) | {t:.timestamp, type, details}'

(Use databricks jobs get-run <run-id> to find the job cluster ID for job runs.)

2. Find what pulls data to the driver

These patterns bring data into driver memory:

Code Problem Replace with
df.collect(), df.toPandas() Whole dataset in the driver Write to a table; aggregate first; limit()
spark.read... then Python loop over rows Driver processes all rows DataFrame transformations / UDF
display(df) on huge DataFrames in jobs Results collected Remove from jobs
Large broadcast() Driver builds the broadcast Remove the hint or reduce the table
Listing millions of files (dbutils.fs.ls recursion) Huge listing held in memory Auto Loader, or narrow paths
Many small Spark jobs in a loop over partitions Query plans and metadata pile up One set-based job

Quick search through the job's notebook or repo:

grep -rnE "collect\(\)|toPandas\(\)|broadcast\(" ./jobs/

3. Get today's run through

Short-term options, quickest first:

  • Bigger driver: set a memory-optimised driver type on the job cluster (for example an E-series VM) while leaving workers unchanged.
  • Raise result limit only if needed: spark.driver.maxResultSize controls how much a collect may return; raising it often just moves the OOM.
  • Turn off automatic broadcast if a broadcast join went wrong after data growth:
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")

Tip

Job clusters can have a different driver node type from workers. Set driver_node_type_id in the job's cluster spec rather than making every worker bigger.

"new_cluster": {
  "spark_version": "15.4.x-scala2.12",
  "node_type_id": "Standard_D8ds_v5",
  "driver_node_type_id": "Standard_E16ds_v5",
  "num_workers": 8
}

4. Fix the code

Replace driver-side processing with distributed operations:

# Before: collects and loops on the driver
rows = df.collect()
for r in rows:
    write_to_api(r)

# After: process per partition on executors
def send_partition(it):
    session = make_session()
    for r in it:
        write_to_api(r, session)
df.foreachPartition(send_partition)

For toPandas() on large data, use the pandas API on Spark (pyspark.pandas) or Arrow-backed mapInPandas, which run on executors.

5. Shared clusters: noisy neighbours

On all-purpose clusters, another user's notebook can exhaust the driver and kill everyone's work. Move scheduled production jobs to job clusters so each run has its own driver.

6. Verify

Rerun the job, watch driver memory in the cluster metrics, and confirm the stage that failed now completes.

Prevent it next time

  • Ban collect()/toPandas() on unbounded data in code review; add a lint rule.
  • Run production on job clusters with a policy that fixes driver and worker types.
  • Alert on job duration and driver restarts, not just failures.
  • Test with production-scale data before releases.

Related fixes