Home / Databricks on Azure

Databricks on AzureTroubleshootingAdvanced

FetchFailedException and ExecutorLostFailure in Azure Databricks: lost shuffle files from OOM, spot evictions and decommissioning

Big jobs retry stages over and over with FetchFailedException or ExecutorLostFailure. Work out whether executors died from memory, spot eviction or node loss, and fix partition sizing, memory, skew and spot settings.

Deepak Bhardwaj5 min read

Applies toAzure Databricks classic job and all-purpose compute (Databricks Runtime 13.3 LTS-16.x), Azure spot VMs, Spark shuffle

Symptoms

  • Stages fail and retry with org.apache.spark.shuffle.FetchFailedException: ... Connection from /10.139.64.12 closed or Failed to connect to ....
  • ExecutorLostFailure (executor 7 exited caused by one of the running tasks) Reason: Remote RPC client disassociated or Executor heartbeat timed out.
  • The same job sometimes takes 40 minutes and sometimes 3 hours, or fails after spark.stage.maxConsecutiveAttempts retries.

A shuffle writes intermediate files on executors' local disks. When an executor (or its VM) disappears, reducers can't fetch those files: FetchFailedException. Spark must recompute the parent stage. One lost node can cause a cascade of retries.

Step 1: why did executors die?

Check the cluster event log first:

databricks clusters events <cluster-id> --output json \
  | jq -r '.events[] | [.timestamp, .type, (.details.reason.code // ""), (.details.reason.parameters // {} | tostring)] | @tsv' \
  | tail -40

(For job clusters, open the run's cluster in the UI, or query system.compute.node_timeline / cluster events in system tables where enabled.)

Event / evidence Cause
Node lost / spot termination events Azure spot eviction
Executor exit codes like 137, "Container killed", OOM messages in executor logs Out of memory
DRIVER_UNAVAILABLE, driver restarts Driver problem (separate issue)
Nodes lost without spot Infrastructure or VM health; retry, open a ticket if frequent

Executor logs (Spark UI > Executors > stderr/stdout of a dead executor, or cluster log delivery) show java.lang.OutOfMemoryError or a killed process when memory is the cause.

Usually caused by a few huge partitions, not overall data size.

In the Spark UI for the failing stage, look at Summary Metrics: max vs median Shuffle Read Size and Spill (disk). If max is 10× the median, it's skew.

Fix How
Let AQE split skewed partitions spark.sql.adaptive.enabled=true and spark.sql.adaptive.skewJoin.enabled=true (defaults on recent runtimes)
More, smaller shuffle partitions spark.sql.shuffle.partitions=auto (Databricks) or a number giving ~100–200 MB per partition
Salt a hot key Add a random suffix to the hot key on the big side, replicate the small side
Avoid exploding joins Check for many-to-many joins multiplying rows
Bigger memory per core Memory-optimised workers (E-series) for wide joins/aggregations
Don't cache huge DataFrames Remove .cache() on data used once 
spark.conf.set("spark.sql.shuffle.partitions", "auto")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "128m")

Step 3: spot evictions

Azure can reclaim spot VMs at any time. Each eviction loses that node's shuffle files.

Use spot with on-demand fallback and an on-demand driver:

"azure_attributes": {
  "availability": "SPOT_WITH_FALLBACK_AZURE",
  "first_on_demand": 1,
  "spot_bid_max_price": -1
}
  • first_on_demand: 1 keeps the driver on an on-demand VM.
  • SPOT_WITH_FALLBACK_AZURE replaces evicted capacity with on-demand VMs when spot isn't available.
  • spot_bid_max_price: -1 pays up to the on-demand price, reducing price-based evictions.

Enable decommissioning so executors on nodes about to be evicted migrate shuffle and cached blocks when Azure gives notice (Databricks supports graceful decommissioning for spot on recent runtimes; check whether it's on by default for your runtime):

spark.decommission.enabled true
spark.storage.decommission.enabled true
spark.storage.decommission.shuffleBlocks.enabled true
spark.storage.decommission.rddBlocks.enabled true

Azure's eviction notice is short, so migration can't save everything on large nodes; it reduces recomputation rather than eliminating it.

Tip

Long jobs with huge shuffles are the worst fit for spot. Run them on on-demand workers, and use spot for short or easily retried jobs. A 3-hour job that loses an hour to recomputation costs more than the spot discount saves.

Step 4: shuffle fetch settings

When executors are alive but busy (long GC, overloaded disks), fetches time out. Moderate increases help:

spark.network.timeout 300s
spark.shuffle.io.maxRetries 10
spark.shuffle.io.retryWait 30s

These hide symptoms if the root cause is memory or skew; fix those first.

Step 5: storage and instance choice

  • Workers with local NVMe/SSD (e.g. d-suffixed sizes like Standard_E16ds_v5) handle shuffle and spill much better than those using only remote disks.
  • Very large nodes concentrate risk: losing one 64-core node loses a lot of shuffle data. Mid-size nodes spread it.
  • Photon reduces memory pressure for many SQL workloads; try it on heavy joins and aggregations.

Decision table

Evidence Fix
OOM in executor logs, skewed tasks AQE skew join, salting, more partitions, memory-optimised nodes
Spot termination events Fallback, on-demand driver, decommissioning, or on-demand for long jobs
Fetch timeouts, no executor deaths GC/disk pressure; local SSD nodes; moderate timeout increases
Random node losses without spot Retry policy on the job; ticket if persistent

Prevent it next time

  • Look at stage summary metrics (max vs median) whenever a job slows down.
  • Set job-level retries for transient failures, but alert on retry counts.
  • Choose spot per job based on runtime and shuffle size.
  • Prefer worker types with local SSD for shuffle-heavy pipelines.

Related fixes