Spark broadcast join failures: 'Could not execute broadcast in 300 secs', driver OOM and the 8 GB limit

#Symptoms

One of these, usually after data growth or a code change:

org.apache.spark.SparkException: Could not execute broadcast in 300 secs.
You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or disable broadcast join by setting spark.sql.autoBroadcastJoinThreshold to -1

java.lang.OutOfMemoryError: Java heap space   (on the driver)
SparkException: Total size of serialized results ... is bigger than spark.driver.maxResultSize
SparkException: Cannot broadcast the table that is larger than 8GB

#How broadcast joins work

For a broadcast hash join, Spark:

1. Runs the query for the small side and collects it to the driver.
2. Builds a hash table and sends it to every executor.
3. Joins each partition of the big side locally, with no shuffle.

So the "small" side must fit in driver memory, serialize within limits, and reach executors before spark.sql.broadcastTimeout (default 300 s). The timeout clock also includes the time to compute the small side, which surprises many people.

#Step 1: confirm the plan

df = orders.join(customers, "customer_id")
df.explain("formatted")

Look for BroadcastHashJoin and BroadcastExchange. In the Spark UI, the SQL tab shows the size of the broadcast relation (data size) and which side was broadcast.

#Step 2: find why Spark thought it was small

Spark broadcasts when the estimated size is below spark.sql.autoBroadcastJoinThreshold (default 10 MB). With AQE, it can also switch to broadcast at runtime when the actual shuffle output is below spark.sql.adaptive.autoBroadcastJoinThreshold.

Common reasons for a wrong decision:

Refresh statistics:

ANALYZE TABLE sales.customers COMPUTE STATISTICS;
-- column stats help join and filter estimates
ANALYZE TABLE sales.customers COMPUTE STATISTICS FOR COLUMNS customer_id, region;

#Step 3: choose the right fix

The small side really is small but slow to compute. Raise the timeout, or materialize it first so the broadcast itself is quick:

spark.conf.set("spark.sql.broadcastTimeout", "900")

The "small" side isn't small anymore. Stop broadcasting it. Use a hint just for this join instead of turning broadcast off everywhere:

SELECT /*+ SHUFFLE_MERGE(c) */ o.*, c.region
FROM sales.orders o JOIN sales.customers c ON o.customer_id = c.customer_id;

Or in PySpark: orders.join(customers.hint("shuffle_merge"), "customer_id"). Remove any broadcast() call that forces the old behaviour.

Global threshold set too high. Put it back near the default and broadcast specific tables with hints where you've measured the benefit:

spark.conf.set("spark.sql.autoBroadcastJoinThreshold", str(50 * 1024 * 1024))  # 50 MB

Broadcast is right, but the driver is too small. A 1-2 GB broadcast needs a driver with plenty of heap. Increase driver memory or node size, and raise spark.driver.maxResultSize only after confirming the table should be broadcast.

Warning

Setting spark.sql.autoBroadcastJoinThreshold = -1 for the whole cluster fixes the failure but can make many other queries slower (more shuffles). Prefer per-join hints.

#Step 4: AQE-specific cases

With AQE on (default in Spark 3.2+), a join planned as sort-merge can be converted to broadcast once Spark sees the actual size. That's usually good. Problems occur when:

• spark.sql.adaptive.autoBroadcastJoinThreshold is set high, so a large runtime side gets broadcast.
• Many concurrent queries on a shared cluster broadcast at once and exhaust driver memory.

Set the AQE threshold explicitly if you've raised the static one, so they don't silently follow each other.

#Quick reference

#Prevent it next time

• Run ANALYZE TABLE as part of table maintenance for dimension tables.
• Review explicit broadcast() hints when tables grow; prefer letting AQE decide.
• Track driver memory usage in your job monitoring; a creeping driver heap is an early warning.
• Related: skew joins, AQE and salting, Databricks driver not responding.