Debugging failed Databricks jobs: OOM, skew and slow stages

#Start with the run output

databricks jobs list-runs --job-id 123 --limit 5 -o json | jq '.[] | {run_id, state: .state.result_state, msg: .state.state_message}'
databricks jobs get-run-output <task_run_id> -o json | jq '.error, .error_trace' -r | head -40

The first line of the error trace tells you which section below to read.

#Driver OOM

Causes, from most to least common:

1. collect(), toPandas() or display() on a large DataFrame.
2. Broadcasting a table that's too big (spark.sql.autoBroadcastJoinThreshold, or a forced broadcast() hint).
3. Many small files or a huge number of partitions, which bloats query planning metadata.

Fixes: write results to a table instead of collecting them, use a larger driver node type, and remove forced broadcasts.

#Executor OOM

# See where the memory goes
spark.conf.get("spark.sql.shuffle.partitions")   # default 200 - often too few for TBs
spark.conf.set("spark.sql.adaptive.enabled", "true")            # on by default in DBR
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")

• Increase shuffle partitions so each task handles roughly 100–200 MB, or let AQE do it.
• Prefer memory-optimized nodes for heavy joins and aggregations.
• Look for explode() and wide window functions, which multiply row counts.

#Data skew

In the Spark UI, open the slow stage, then Summary Metrics. If the max task duration or shuffle read is more than 10× the median, you have skew.

# Which keys are hot?
df.groupBy("customer_id").count().orderBy("count", ascending=False).show(10)

Fixes:

• AQE skew join handling: spark.sql.adaptive.skewJoin.enabled=true (on by default).
• Filter out or separately process null/default keys. These are often the "hot" key.
• Salt the key for aggregations where AQE can't help.

#Repair instead of rerun

For multi-task jobs, Repair run in the UI (or databricks jobs repair-run) reruns only the failed tasks and the tasks downstream of them. That saves time and cost on long pipelines.

#Make failures easier to debug next time

• Enable cluster log delivery to DBFS/Volumes/cloud storage so driver and executor logs outlive the cluster.
• Set retries with a back-off on tasks that hit flaky external sources, and don't set retries on deterministic code errors.
• Alert on job duration as well as failure. A job that slowly gets 3× longer is usually skew building up.

Note

Serverless jobs don't expose the full Spark UI. Use the query profile to see the same per-operator time and row counts.