Home / Databricks on Azure

Databricks on AzureRunbookIntermediate

Sev1: a Databricks Asset Bundle deploy broke production jobs — roll back fast and safely

After a bundle deployment, production jobs fail or run the wrong code. Identify what changed, redeploy the last good commit, handle deleted jobs and schedules, and repair failed runs.

Deepak Bhardwaj3 min read

Applies toAzure Databricks, Databricks Asset Bundles, Databricks CLI 0.2xx, Git-based CI/CD

Symptoms

  • Many jobs fail right after a CI/CD release, or run with wrong parameters, clusters or notebooks.
  • Some jobs disappeared, or schedules are paused.
  • Someone deployed to the prod target from the wrong branch.

1. Stop further damage

Pause the CI/CD pipeline so no new deploy runs while you fix it. Then check which jobs are failing:

databricks jobs list-runs --active-only=false --limit 50 --output json \
  | jq -r '.[] | select(.state.result_state=="FAILED") | "\(.job_id) \(.run_name) \(.state.state_message)"'

2. Find what changed

Bundles record deployment state in the workspace (by default under the deploying user's or service principal's .bundle/<name>/<target> folder). Compare the last good commit with the deployed one:

git log --oneline -10 -- databricks.yml resources/
git diff <last-good-sha> <bad-sha> -- databricks.yml resources/

Common breaking changes:

Change Effect
Job removed from resources deploy deletes the job from the workspace
mode: development used for prod target Jobs prefixed with user name, schedules paused
Changed run_as Jobs run as an identity without data access
New cluster policy or node type Clusters fail to start
Wildcard include missed files Resources missing

3. Roll back by redeploying the last good commit

git checkout <last-good-sha>
databricks bundle validate -t prod
databricks bundle deploy -t prod

Deploy from the CI service principal, not a personal account, so ownership and run_as stay correct. The deployment updates existing jobs in place (same job IDs) and recreates jobs that were deleted (with new job IDs).

Warning

Recreated jobs get new IDs. Anything that triggers jobs by ID (ADF pipelines, external schedulers, run_job_task in other jobs) must be updated. Bundle-managed references using ${resources.jobs.<name>.id} update automatically.

4. Check schedules and permissions

databricks jobs get <job-id> --output json | jq '{name: .settings.name, schedule: .settings.schedule, run_as: .run_as_user_name}'
  • Schedules must be UNPAUSED in production. In mode: production the bundle respects the schedule in the YAML; in development mode schedules are paused.
  • Check job permissions (CAN_MANAGE_RUN for operators) came back with the redeploy.

5. Repair failed runs

For multi-task jobs, repair only the failed tasks instead of rerunning everything:

databricks jobs repair-run <run-id> --rerun-all-failed-tasks

Check that tasks which may have partly written data are idempotent before repairing. For Delta targets, DESCRIBE HISTORY shows whether the failed run committed anything.

6. Fix forward

Once stable, fix the bad change on a branch, deploy to a staging target, run the jobs there, then promote.

targets:
  prod:
    mode: production
    workspace:
      host: https://adb-1234567890123456.7.azuredatabricks.net
    run_as:
      service_principal_name: "<sp-application-id>"

Prevent it next time

  • Require pull requests and protected branches for the production target.
  • Deploy to staging and run a smoke job before prod in the same pipeline.
  • Always set mode: production and run_as explicitly for prod.
  • Keep the last good commit SHA recorded in the release, so rollback is one command.

Related fixes