Repair a failed Databricks job run without duplicating data: repair-run, latest_repair_id and dependent tasks

#Symptom

A multi-task job failed at task 6 of 10. Tasks 1 to 5 took two hours and succeeded. You want to fix the problem and continue from the failure, not start again.

Repair does that. It reruns the unsuccessful tasks and their dependents inside the same job run, and it leaves successful tasks alone. Three things catch people out:

• Each repaired task restarts from the beginning. If it wrote part of its output before failing, rerunning it can duplicate that data.
• Repaired tasks use the current job and task settings, not the settings from the original run.
• Repair only works on jobs with two or more tasks. For a single-task job, use Run now.

Caution

Databricks doesn't make tasks idempotent for you. Check what the failed task wrote before you repair it (Step 2). A repair of an append-only task can load the same data twice.

#Step 1: find the run and see which tasks failed

List recent runs of the job. The CLI has no result-state filter, so filter the output:

databricks jobs list-runs --job-id 123 --completed-only --limit 25 -p prod -o json \
  | jq '.[] | select(.state.result_state != "SUCCESS")
        | {run_id, result: .state.result_state, msg: .state.state_message}'

Then look at the tasks and any earlier repairs:

databricks jobs get-run 456789 --include-history -p prod -o json > run.json

jq '.tasks[] | {task_key, run_id, attempt_number, result: .state.result_state}' run.json
jq '.repair_history' run.json

The state field is deprecated in favor of status, but its result states are the easiest to read here. Task and repair arrays page at 100 items.

Get the error using the task's run_id, not the parent run ID:

databricks jobs get-run-output 456801 -p prod -o json | jq -r '.error, .error_trace'

The output is limited to the first 5 MB.

#Step 2: check for partial writes

Before repairing, find out whether the failed task changed its target. Delta history shows each commit and the job behind it:

DESCRIBE HISTORY prod.sales.orders LIMIT 20;

Look at timestamp, operation, job, and operationMetrics such as numOutputRows. Compare the commits with the failed task's start and end time.

If the task committed a partial append, you can roll the table back:

-- Confirm the version in DESCRIBE HISTORY first
RESTORE TABLE prod.sales.orders TO VERSION AS OF 1841;

Warning

RESTORE also removes commits that other jobs made after that version. If anything else writes to the table, delete only the affected rows instead, for example with DELETE FROM ... WHERE load_date = '2026-11-03'. Confirm the rows belong only to this load.

#Step 3: check what the repair will run

Repair uses current settings. Before you start, find out whether anyone changed the job after the failure:

databricks jobs get 123 -p prod -o json > job-now.json
jq '.settings.tasks[] | {task_key, depends_on, job_cluster_key}' job-now.json

• A deployment since the failure means the repaired tasks run the new configuration. That helps if it contains your fix. It's a risk if it brings in unrelated changes.
• If tasks share a job cluster, the repair creates a new one. my_job_cluster becomes my_job_cluster_v1 with the current settings. Don't expect anything left on the original cluster's local disk, or any state cached there, to be available.
• Task disabled state doesn't decide what gets repaired. Repairs use each task's run state. To force a disabled task to run, include it in rerun_tasks.

#Step 4: run the repair

The run must not be in progress. Choose the narrowest option that does the job.

The common case:

databricks jobs repair-run 456789 --rerun-all-failed-tasks --no-wait -p prod

Without --no-wait, the CLI waits up to 20 minutes by default. Use --no-wait for long pipelines, or set --timeout.

To name tasks, put the body in a file. This avoids JSON quoting problems in PowerShell. With --json, the run ID goes in the body, not on the command line.

{
  "run_id": 456789,
  "rerun_tasks": ["load_silver_orders"],
  "rerun_dependent_tasks": true,
  "job_parameters": { "load_date": "2026-11-03" }
}

databricks jobs repair-run --json @repair.json --no-wait -p prod

Warning

rerun_dependent_tasks also reruns downstream tasks that already succeeded. Apply the Step 2 checks to each of them before you use it.

In the UI, open the job run and click Repair run. The dialog lists unsuccessful and dependent tasks. Parameters you enter override existing values. To go back to the original value on a later repair, clear the key and value.

#Step 5: repair the same run again

A second repair of the same run must pass the ID of the latest repair. Read it from repair_history:

LATEST=$(databricks jobs get-run 456789 --include-history -p prod -o json \
  | jq '[.repair_history[] | select(.type == "REPAIR")] | last | .id')

databricks jobs repair-run 456789 --rerun-all-failed-tasks \
  --latest-repair-id "$LATEST" --no-wait -p prod

#Step 6: verify

databricks jobs get-run 456789 --include-history -p prod -o json \
  | jq '{result: .state.result_state,
         tasks: [.tasks[] | {task_key, result: .state.result_state}],
         repairs: [.repair_history[] | {type, id, start_time, end_time}]}'

Then check the data:

-- Duplicate check on the business key for the repaired load
SELECT order_id, COUNT(*) AS copies
FROM prod.sales.orders
WHERE load_date = '2026-11-03'
GROUP BY order_id
HAVING COUNT(*) > 1
LIMIT 20;

The run's reported duration includes everything from the first start to the end of the latest repair. Expect the run to show a long duration. Repaired tasks stay part of the original job run, so use the same run ID when you check it.

#Prevent it next time

• Make writes safe to repeat. Use MERGE on a business key, or overwrite only the load window:

(df.write.format("delta")
   .mode("overwrite")
   .option("replaceWhere", f"load_date = '{load_date}'")
   .saveAsTable("prod.sales.orders"))

• Pass the load date as a job parameter so a repair processes the same window. Avoid current_date() inside task logic, because a repair the next day would load a different window.
• Use task retries for transient failures, and keep repair for failures that need a person to act.
• Split long tasks into smaller ones, so a repair repeats less work.
• Export run results you need for audits. Runs are removed automatically after 60 days.