Sev1: Databricks job ran twice at the same time — duplicates, conflicts and how to stop it

#Symptoms

• Two active runs of one job; downstream tables have duplicate rows.
• One run fails with ConcurrentAppendException / ConcurrentDeleteReadException on a Delta table.
• An ADF pipeline shows a timeout and a retry while the first Databricks run was still going.

#1. Where did the second run come from?

databricks jobs list-runs --job-id 123 --limit 10 --output json | \
  jq -r '.[] | [.run_id, .trigger, (.start_time/1000|todate), .state.life_cycle_state, .state.result_state] | @tsv'

ADF's Databricks Notebook / Job activity with a timeout and retries: when the activity times out, ADF may cancel or abandon the run and start another; if cancellation didn't reach Databricks, both run.

#2. Stop the bleeding

Cancel the extra run:

databricks jobs cancel-run <run-id>

Pause the schedule or trigger until the fix is in.

#3. Clean the data

Use Delta history to see both runs' writes:

DESCRIBE HISTORY main.silver.orders LIMIT 10;

If the table was clean before both runs, restore to the version before them and rerun once. Otherwise deduplicate on the business key (see the duplicates runbook).

#4. Enforce one run at a time

{
  "max_concurrent_runs": 1,
  "queue": { "enabled": true }
}

With max_concurrent_runs: 1 and queueing on, a second trigger waits instead of running in parallel (without queueing it's skipped). Set it in the job or bundle definition:

resources:
  jobs:
    orders_load:
      max_concurrent_runs: 1
      queue:
        enabled: true

For ADF:

• Set the activity timeout comfortably above the job's normal runtime, and retry = 0 for non-idempotent jobs.
• Prefer the Databricks Job activity (triggers an existing job) so the job's own concurrency setting applies.

#5. Make writes idempotent anyway

Concurrency limits reduce risk; idempotent writes remove it:

• MERGE on business keys, or INSERT OVERWRITE ... REPLACE WHERE for partition loads.
• Delta row-level concurrency (with deletion vectors / liquid clustering on recent runtimes) reduces write conflicts for concurrent MERGEs on different rows.

Tip

Tag every written row with the job run ID ({{job.run_id}} as a parameter). When duplicates appear, you can delete exactly one run's rows.

#Prevent it next time

• max_concurrent_runs: 1 + queue on every production job that writes tables.
• Alert when a run's duration exceeds the schedule interval.
• One orchestrator per job: don't schedule in both ADF and Databricks.