Applies toDatabricks Runtime 13.3 LTS+ (Spark 3.4+), jobs and notebooks on classic or serverless compute
1. Measure before changing
- Spark UI (classic) or query profile (serverless/SQL): longest stages, shuffle read/write, spill, task time distribution.
- Job run duration trend from
system.lakeflow.job_run_timeline. - Pick the top 5 jobs by DBU (cost post) — tuning those matters most.
2. Usual problems and fixes
| Symptom | Cause | Fix |
|---|---|---|
| A few tasks run far longer than the rest | Skew | AQE skew join (on by default); salt keys; filter nulls before joins |
| Huge shuffle, many stages | Wide joins/aggregations on unfiltered data | Filter and select columns early; pre-aggregate |
Spill (disk) large | Not enough memory per task | More shuffle partitions (less data per task) or memory-optimised nodes |
| Thousands of tiny tasks | Too many small files | Compaction / liquid clustering; OPTIMIZE |
| Slow Python logic row by row | Python UDFs | Built-in functions, or pandas UDFs (vectorised) |
| Broadcast timeouts / OOM on driver | Broadcasting a large table | Raise or disable the auto-broadcast threshold for that query |
| Same DataFrame recomputed | No reuse | cache() only when reused and it fits; unpersist afterwards |
| Driver busy, executors idle | collect(), toPandas(), loops on the driver | Keep work distributed |
3. Adaptive Query Execution
On by default. Useful settings:
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.shuffle.partitions", "auto") # Databricks: auto-optimised shuffle
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
Don't hard-code spark.sql.shuffle.partitions = 200 copied from old code — on Databricks auto usually does better.
4. Joins
from pyspark.sql.functions import broadcast
result = facts.join(broadcast(dim_small), "customer_id")
- Broadcast small dimensions explicitly when the optimizer doesn't.
- Join on columns with the same type (implicit casts block optimisations).
- Filter both sides before the join; use dynamic file pruning by joining on clustered/partitioned columns.
5. File layout
- Target file sizes of roughly 100 MB–1 GB; avoid millions of small files.
- Liquid clustering on the columns used in filters and joins (see the table optimisation post).
- Partition only very large tables, on low-cardinality columns (date), if at all.
6. Photon
Photon is a vectorised engine: big gains on SQL/DataFrame scans, joins, aggregations and Delta writes; little gain on Python UDF-heavy or ML code. It has a higher DBU rate, so compare:
cost = runtime × DBU rate × nodes
Run the job once with and once without Photon (same cluster size) and keep the cheaper one.
7. Right compute
- Memory-optimised nodes for heavy joins and wide aggregations; compute-optimised for CPU-bound transforms.
- Enable autoscaling for variable stages; fixed size for predictable batch.
- For short jobs, serverless removes cluster start time.
Tip
Change one thing at a time and record runtime and DBUs per change. Tuning by intuition often makes a job faster and more expensive at the same time.