Databricks job runs queued or skipped on Azure: max concurrent runs, workspace limits and clearing a backlog

#Symptom

A scheduled run didn't do any work, and no task failed. You see one of these:

• The run sits in Queued while downstream data goes stale.
• The run shows as Skipped, sometimes for hours of consecutive schedules.
• Runs from many unrelated jobs queue at the same time.
• A trigger or API call returns 429 Too Many Requests.

These runs usually aren't failures, so failure alerts may stay silent.

#Which limit did you hit?

JOB_ID=123456

# Current runs, including queued ones
databricks jobs list-runs --job-id "$JOB_ID" --active-only -o json | \
  jq '.[] | {run_id, start_time, state: (.status.state // .state.life_cycle_state), queue: .status.queue_details}'

# The job's concurrency and queue settings
databricks jobs get "$JOB_ID" -o json | \
  jq '.settings | {max_concurrent_runs, queue, schedule}'

# Why one finished run ended the way it did
databricks jobs get-run <run-id> -o json | jq '.status'

Older CLI versions may return only the deprecated state object, so upgrade if status is missing.

#How the settings interact

• max_concurrent_runs defaults to 1 and can't exceed 1000. Setting 0 skips all new runs, which is an easy way to leave a job silently disabled after an incident.
• With queueing on, a run that hits a limit waits for up to 48 hours. With queueing off, a run that hits the job's limit is skipped.
• Queueing is on by default only for jobs created in the UI after April 15, 2024. Jobs created earlier or from the API, Terraform or bundles may have it off, so check the setting rather than assume.
• Queueing is set per job. Enabling it on one job doesn't protect the others.

#The usual root cause: one stuck run

With max_concurrent_runs: 1, a single hung run blocks every later schedule. Check how long the active run has been going:

databricks jobs list-runs --job-id "$JOB_ID" --active-only -o json | \
  jq -r '.[] | [.run_id, (.start_time/1000 | todate), (.status.state // .state.life_cycle_state)] | @tsv'

If the oldest run is far past its normal duration, that's the incident to fix before you change any limits.

#Current workspace limits

Fixed limits at the time of writing (check the current limits, as these change):

#Find the pattern with system tables

Runs dropped in the past week, by job and reason:

SELECT workspace_id, job_id, termination_code,
       COUNT(*) AS runs, MAX(period_start_time) AS latest
FROM system.lakeflow.job_run_timeline
WHERE period_start_time >= current_timestamp() - INTERVAL 7 DAYS
  AND result_state IS NOT NULL
  AND termination_code IN ('SKIPPED', 'MAX_CONCURRENT_RUNS_EXCEEDED',
      'WORKSPACE_RUN_LIMIT_EXCEEDED', 'CLUSTER_REQUEST_LIMIT_EXCEEDED',
      'MAX_JOB_QUEUE_SIZE_EXCEEDED')
GROUP BY workspace_id, job_id, termination_code
ORDER BY runs DESC;

The busiest scheduled start minutes, which show a top-of-the-hour stampede:

WITH starts AS (
  SELECT workspace_id, run_id, MIN(period_start_time) AS started
  FROM system.lakeflow.job_run_timeline
  WHERE period_start_time >= current_timestamp() - INTERVAL 7 DAYS
    AND trigger_type = 'CRON'
  GROUP BY workspace_id, run_id
)
SELECT workspace_id, date_trunc('MINUTE', started) AS start_minute, COUNT(*) AS runs
FROM starts
GROUP BY workspace_id, date_trunc('MINUTE', started)
ORDER BY runs DESC
LIMIT 20;

A few notes on reading these results:

• result_state is populated only on a run's final row. Runs longer than an hour are split into hourly rows, so aggregate by run_id.
• A run that never started has period_start_time equal to period_end_time.
• queue_duration_seconds only has values for single-task legacy API jobs. It logs 0 for other runs, so don't use it to rule out queueing.
• System tables lag behind live data. During an incident, use the CLI output above.

#Clear a backlog safely

First decide whether the queued runs still need to run. If each run processes the latest data, one run is enough, and the rest just waste compute. If each run has its own parameters, such as a date, cancelling them loses work unless you re-trigger those dates.

Cancel only the queued runs of one job, one at a time:

databricks jobs list-runs --job-id "$JOB_ID" --active-only -o json | \
  jq -r '.[] | select((.status.state // .state.life_cycle_state) == "QUEUED") | .run_id' | \
  while read -r RUN_ID; do
    echo "Cancelling queued run $RUN_ID"
    databricks jobs cancel-run "$RUN_ID" --no-wait
  done

Caution

Be careful with databricks jobs cancel-all-runs. It cancels active runs, not just queued ones. With --all-queued-runs and no --job-id, it can cancel queued runs across the whole workspace. Always pass --job-id, and prefer the loop above when a run is mid-write.

Warning

Cancelling a run that's writing data can leave a partial load. Delta commits are atomic per write, but a multi-step job can stop between steps. Check that your tasks are safe to re-run before you cancel anything in the RUNNING state.

Release the queue gradually. If a shared source was down, hundreds of released runs can hit the cluster request rate limit and fail again.

#Fix the settings

Set the limit and enable queueing:

databricks jobs update --json '{
  "job_id": 123456,
  "new_settings": {
    "max_concurrent_runs": 1,
    "queue": {"enabled": true}
  }
}'

Raise max_concurrent_runs above 1 only if overlapping runs are safe. Two runs loading the same target table can duplicate data or cause write conflicts.

Move schedules off the top of the hour. This example starts at 02:07 UTC:

databricks jobs update --json '{
  "job_id": 123456,
  "new_settings": {
    "schedule": {
      "quartz_cron_expression": "0 7 2 * * ?",
      "timezone_id": "UTC"
    }
  }
}'

Tip

If the job is managed by Terraform or Asset Bundles, make the change there too. Otherwise the next deploy puts the old value back.

Clients that trigger runs through the API should retry 429 responses with exponential backoff and jitter.

#Prevent it next time

• Set timeout_seconds on every job. A timeout stops a hung run from blocking the queue indefinitely.
• Alert on duration and on missing outcomes. Use a duration threshold notification, and a scheduled query on system.lakeflow.job_run_timeline for the termination codes above.
• Enable queueing on every production job, and set it in your job templates so that API-created jobs get it too.
• Stagger schedules. Give each team a minute offset instead of 0 0 * * * ? for everything.
• Limit nested fan-out. Run job and For each tasks have their own lower workspace limit.
• Review max_concurrent_runs: 0 after incidents.