Symptoms
- A job run shows
RUNNINGfar beyond its normal duration; no failure alert fired. - Downstream jobs triggered by its completion haven't started; reports are late.
1. Find which task is stuck
databricks jobs get-run <run-id> --output json \
| jq '.tasks[] | {task: .task_key, state: .state.life_cycle_state, start: .start_time, cluster: .cluster_instance.cluster_id}'
Open the running task's Spark UI:
| What you see | Likely cause |
|---|---|
| Active stage with one or a few tasks running for hours | Data skew (see the skew runbook) |
| No active Spark jobs, cell still running | Driver-side Python: API call without timeout, time.sleep loop, waiting on a lock |
| Stage keeps retrying | Executor loss, shuffle fetch failures |
Cluster still PENDING | Cluster can't start: quota, subnet IPs, pool exhausted |
2. Driver-side hangs
Look at the driver's thread dump (Spark UI > Executors > driver > Thread Dump) or the driver log. Common culprits:
- HTTP calls with no timeout (
requests.get(url)withouttimeout=). - JDBC reads waiting on a database lock.
while not exists(path): sleep(60)waiting for a file that never comes.
Fix the code to always use timeouts:
import requests
r = requests.get(url, timeout=(5, 60)) # connect, read
For JDBC sources, set query timeouts:
(spark.read.format("jdbc")
.option("url", jdbc_url)
.option("queryTimeout", 1800)
...)
3. Cancel and repair
databricks jobs cancel-run <run-id>
# after fixing, rerun only failed/cancelled tasks
databricks jobs repair-run <run-id> --rerun-all-failed-tasks
Warning
A cancelled task that writes with
mode("append")may have written part of its output. Delta commits are atomic per write, but a task doing several writes can stop between them. CheckDESCRIBE HISTORYon the targets before repairing, to avoid duplicates.
4. Unblock downstream
If downstream jobs depend on this one through a job trigger or a parent job, and business agrees to use yesterday's data for one dependency, run downstream tasks manually and record it. Otherwise, repair and let the chain continue.
5. Add guard rails
Set a timeout and a duration warning on every production task:
{
"task_key": "load_orders",
"timeout_seconds": 5400,
"max_retries": 1,
"min_retry_interval_millis": 300000,
"health": {
"rules": [
{"metric": "RUN_DURATION_SECONDS", "op": "GREATER_THAN", "value": 3600}
]
},
"email_notifications": {"on_duration_warning_threshold_exceeded": ["data-oncall@example.com"]}
}
Job-level timeout_seconds caps the whole run. The health rule sends a warning before the timeout, so on-call can act early.
For streaming tasks, use the streaming backlog health metrics instead of duration (check what your workspace supports).
6. Watch for queued runs
If max_concurrent_runs is 1 and a run hangs, the next scheduled runs are skipped or queued (depending on the queue setting). After the fix, check whether you need a catch-up run.
Prevent it next time
timeout_secondsand a duration health rule on every production job.- Timeouts on every external call inside notebooks.
- A daily report of runs longer than P95 duration.
- Alerts routed to the on-call channel, not just personal email.