Home / Data Platform / Databricks & Spark

Data PlatformHow-toIntermediate

Reading the Spark UI to explain a slow job to a customer

Where to look in the Spark UI (jobs, stages, task metrics, SQL plans) to find skew, spills, small files and bad joins, with the fix for each pattern.

Deepak Bhardwaj3 min read

Applies toApache Spark 3.x, Databricks, Synapse Spark, EMR, Fabric Spark

When a customer says "the job used to take 20 minutes, now it takes 2 hours", the Spark UI almost always shows why. Here's where to look, in order.

1. Jobs and stages: where did the time go?

Open Jobs, sort by Duration, and click the slowest job, then its slowest stage. Usually one stage accounts for most of the runtime.

2. Stage summary metrics

On the stage page, look at Summary Metrics for Completed Tasks:

Pattern Meaning
Max duration ≫ median (10×+) Skew: a few tasks process most of the data
Spill (Memory) / Spill (Disk) above 0 Partitions don't fit in memory. Use more partitions or bigger executors.
High Shuffle Read Size everywhere Wide shuffles. Check the join strategy.
Thousands of tiny tasks of a few ms Too many partitions or small files
Long Scheduler Delay / Task Deserialization Driver overloaded, or a huge closure (a big broadcast variable)
High GC Time (> 10% of task time) Executor memory pressure

3. SQL / DataFrame tab: the physical plan

Open the query, then Details. Things to check:

  • Join type: SortMergeJoin on a small table that should be BroadcastHashJoin. Small tables over spark.sql.autoBroadcastJoinThreshold (10 MB by default) won't broadcast.
  • Scan nodes: number of files read and files pruned. If filters on partition columns don't prune, the filter is probably on a transformed column (to_date(ts) instead of the partition column).
  • AQE notes: with Adaptive Query Execution, look for AQEShuffleRead and the "skew" markers that show AQE split skewed partitions.

4. Executors tab

  • Dead executors with exit code 137 or 143 → killed for memory. Raise memory overhead, or reduce partition size.
  • Uneven input per executor → skew, or data locality problems.

Fixes by pattern

# Skew: let AQE handle it, or salt the hot key
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

# Too many small partitions after a filter
df = df.coalesce(64)

# Spills: more, smaller shuffle partitions (or let AQE coalesce)
spark.conf.set("spark.sql.shuffle.partitions", "800")

# Force a broadcast of a known-small dimension
from pyspark.sql.functions import broadcast
fact.join(broadcast(dim), "customer_id")

What changed?

If the job got slower without any code change, compare with an older run:

  • Input size: did the data grow, or did a new partition become huge (often a backlog or a reprocessing day)?
  • File count: a new upstream writer producing thousands of small files.
  • Cluster or runtime change: a smaller node type, Spot capacity loss, or a new runtime with a different default.

Tip

Save the Spark event logs (cluster log delivery). Without them, the Spark UI of a finished job on a terminated cluster is gone, and so is your evidence.

Related fixes