Sev1: dozens of Databricks jobs fail at once after a cluster policy change

#Symptoms

• Many unrelated jobs fail at cluster creation, often at the same scheduled hour.
• Run error messages such as:
  • INVALID_PARAMETER_VALUE: Validation failed for spark_version, the value must be one of [...]
  • Validation failed for node_type_id, the value must be ...
  • Validation failed for autotermination_minutes / custom_tags.CostCenter is required
• Interactive users may also be unable to start or edit clusters.

#Step 1: confirm it is the policy (5 minutes)

databricks jobs list-runs --active-only=false --limit 50 -o json \
  | jq -r '.[] | select(.state.result_state=="FAILED") | [.job_id, .run_id, .state.state_message] | @tsv' | head -30

Group the messages; a common Validation failed for <attribute> points straight at the rule.

Which policy changed and when:

databricks cluster-policies list -o json | jq -r '.[] | [.policy_id, .name] | @tsv'
databricks cluster-policies get <policy-id> -o json | jq -r '.definition' | jq .

The audit log (system table) shows who edited it:

SELECT event_time, user_identity.email, action_name, request_params
FROM system.access.audit
WHERE service_name = 'clusterPolicies' AND event_time > current_timestamp() - INTERVAL 2 DAYS
ORDER BY event_time DESC;

#Step 2: restore service (15 minutes)

Option A: roll back the policy definition. If the old definition is in Git (it should be), re-apply it:

databricks cluster-policies edit --json '{
  "policy_id": "<policy-id>",
  "name": "job-compute-standard",
  "definition": "<old definition JSON as an escaped string>"
}'

Without Git, reconstruct it from the audit log request_params of the previous edit.

Option B: relax just the failing rule. For example, a policy now forces a newer runtime that some jobs pinned explicitly:

"spark_version": {
  "type": "allowlist",
  "values": ["15.4.x-scala2.12", "16.4.x-scala2.12", "14.3.x-scala2.12"],
  "defaultValue": "15.4.x-scala2.12"
}

Or a new required tag that jobs do not set. Give it a default so jobs pass validation:

"custom_tags.CostCenter": { "type": "fixed", "value": "CC-UNASSIGNED" }

Warning

A fixed value overrides what the job asks for, while allowlist/regex without a default rejects jobs that do not set it. When tightening a policy, prefer adding a default first and enforcing later.

#Step 3: rerun the failed jobs

Repair runs re-run only failed tasks:

for run in $(databricks jobs list-runs --limit 100 -o json | jq -r '.[] | select(.state.result_state=="FAILED" and .start_time > '"$(($(date +%s) - 43200))"'000) | .run_id'); do
  databricks jobs repair-run $run --rerun-all-failed-tasks
done

Prioritise by business impact (finance closes, regulatory feeds first). Check concurrency so 50 repairs at once do not hit compute quota:

az vm list-usage -l westeurope -o table | grep -i "Total Regional vCPUs\|DDSv5"

#Step 4: find jobs that will fail next

Before the next schedule, test every job's cluster spec against the current policy. List job cluster specs that use the policy:

databricks jobs list -o json | jq -r '.[].job_id' | while read id; do
  databricks jobs get $id -o json | jq -r --arg id "$id" \
    '.settings.job_clusters[]?.new_cluster | select(.policy_id=="<policy-id>") | [$id, .spark_version, .node_type_id] | @tsv'
done

Compare the output with the policy's allowed values and fix job definitions (ideally in their asset bundles) that still violate the rules.

#Step 5: confirm

• Failed jobs repaired and succeeded.
• A test job using the policy creates a cluster successfully.
• Next scheduled window runs clean.

#A safer policy change process

1. Policies in Git, deployed by Terraform or bundles.
2. For any change, run a script that validates every existing job cluster spec referencing the policy (as in step 4) and lists violations.
3. Announce the change with the list of affected jobs and owners; give them a deadline.
4. Introduce new requirements as defaults first, enforcement later.
5. Apply outside the main batch window, with someone watching the next runs.

#Common rules that break jobs

#Prevent it next time

• Policy changes reviewed with a list of impacted jobs generated by script.
• Alert when more than N job runs fail with a validation error within an hour.
• Keep policy families (Databricks-provided templates) as the base and override only what you need.