Databricks job alerts that reach on-call - failure, duration and streaming backlog notifications

#The goal

A job fails at 02:00 and nobody knows until the business asks why the dashboard is empty. Or a streaming job stays "Running" for three days while it processes nothing. Both happen because the job has no alerting, or alerting goes to one person's inbox.

This guide sets up four things on every production job:

1. A failure alert to a team destination (Teams, PagerDuty, webhook), not to a person.
2. A duration warning for runs that are slower than normal, plus a hard timeout.
3. A streaming backlog alert for continuous or long-running streams.
4. Noise filters so retries, skipped runs and canceled runs don't page anyone.

Then it shows how to audit the workspace for jobs that have no failure alert at all.

#1. Create team notification destinations

Destinations for Slack, Microsoft Teams, PagerDuty and generic HTTP webhooks are created by a workspace admin under Settings → Notifications → Manage (notification destinations). Jobs reference them by ID.

List the destinations that already exist:

databricks notification-destinations list -p prod

Note the id of each destination. You need it in the job JSON.

Tip

Point production alerts at a shared channel or an on-call service, never at an individual's email. When that person leaves, alerts go nowhere and nobody notices.

The content of Teams and Slack messages can change between releases. If you parse alerts in another system (ITSM, event router), use a webhook destination. Webhook payloads carry event_type (for example jobs.on_failure), workspace_id, run.run_id and job.job_id, which is enough to build a link back to the run.

#2. Add failure, duration and timeout settings to a job

You need CAN MANAGE or IS OWNER on the job. In the UI: open the job, Job details → Job notifications → Edit notifications, and Duration and streaming backlog thresholds for the limits.

For jobs managed as code, these are the relevant job settings (Jobs API / bundle YAML use the same names):

{
  "timeout_seconds": 10800,
  "health": {
    "rules": [
      { "metric": "RUN_DURATION_SECONDS", "op": "GREATER_THAN", "value": 5400 }
    ]
  },
  "email_notifications": {
    "on_failure": ["data-platform-oncall@contoso.com"],
    "on_duration_warning_threshold_exceeded": ["data-platform-oncall@contoso.com"]
  },
  "webhook_notifications": {
    "on_failure": [{ "id": "<teams-destination-id>" }],
    "on_duration_warning_threshold_exceeded": [{ "id": "<teams-destination-id>" }]
  },
  "notification_settings": {
    "no_alert_for_skipped_runs": true,
    "no_alert_for_canceled_runs": true
  }
}

What each piece does:

Apply it to an existing job with a partial update, so you don't overwrite the tasks:

databricks jobs get 123456789 -p prod > job-before.json   # keep a copy

databricks jobs update -p prod --json '{
  "job_id": 123456789,
  "new_settings": {
    "timeout_seconds": 10800,
    "health": { "rules": [ { "metric": "RUN_DURATION_SECONDS", "op": "GREATER_THAN", "value": 5400 } ] },
    "webhook_notifications": {
      "on_failure": [ { "id": "<teams-destination-id>" } ],
      "on_duration_warning_threshold_exceeded": [ { "id": "<teams-destination-id>" } ]
    },
    "notification_settings": { "no_alert_for_skipped_runs": true, "no_alert_for_canceled_runs": true }
  }
}'

Warning

databricks jobs reset replaces the whole job definition. If you send only the notification block with reset, you delete the tasks, schedule and compute. Use update with new_settings for small changes, and keep the jobs get output as a rollback copy.

#Picking the duration threshold

Use the job's own history, not a guess. Take the 95th percentile of successful runs over 30 days and add some margin:

SELECT job_id,
       percentile(timestampdiff(SECOND, period_start_time, period_end_time), 0.95) AS p95_secs,
       max(timestampdiff(SECOND, period_start_time, period_end_time))             AS max_secs,
       count(*) AS runs
FROM system.lakeflow.job_run_timeline
WHERE job_id = '123456789'
  AND result_state = 'SUCCEEDED'
  AND period_start_time >= current_date() - INTERVAL 30 DAYS
GROUP BY job_id;

Long runs can span several rows in this table (it's split into time slices). For a rough threshold this is fine; for exact run durations, group by run_id first and take min(period_start_time) and max(period_end_time).

A common pattern: warning at about 1.5 × p95, timeout at 2–3 × p95. Set the timeout above the longest legitimate run (month-end loads, backfills), or the timeout will kill good runs.

#3. Retries: job-level vs task-level alerts

This catches many teams out:

• Job-level notifications are not sent when a failed task is retried. You get one alert when the run finally fails.
• Task-level notifications fire for each task failure. If a task retries three times, you can get several alerts.

For tasks with retries, set the task notification to Mute notifications until the last retry (UI), so on-call gets one page per real failure. Use task-level alerts when one task has a different owner (for example, an export task owned by another team).

A run that finishes as Succeeded with failures counts as a success. If some tasks are allowed to fail (Run if dependencies), add an on_success notification to a low-priority channel so those partial failures still get seen.

#4. Streaming backlog alerts

A stream that is running but not keeping up never fails. Duration alerts don't help either, because a continuous job has no normal end. Use backlog thresholds instead (Public Preview at the time of writing):

Which metrics are available depends on the source. Check the task's streaming metrics in the run page.

{
  "health": {
    "rules": [ { "metric": "STREAMING_BACKLOG_SECONDS", "op": "GREATER_THAN", "value": 900 } ]
  },
  "webhook_notifications": {
    "on_streaming_backlog_exceeded": [ { "id": "<pagerduty-destination-id>" } ]
  }
}

How it behaves:

• The alert fires when the 10-minute average is above the threshold, then repeats every 30 minutes while it stays high.
• It doesn't work for pipeline tasks (Lakeflow Declarative Pipelines). Use the pipeline's event log or pipeline notifications for those.
• The Jobs service has to see the active streaming query. Don't block the notebook with query.awaitTermination() in a job task; see the Databricks docs on awaitTermination() in production.

#5. Find production jobs with no failure alert

Run this with the Databricks SDK for Python (pip install databricks-sdk) from a laptop or notebook, using a profile with read access to the jobs:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(profile="prod")

for base in w.jobs.list():
    s = w.jobs.get(job_id=base.job_id).settings
    email = s.email_notifications.on_failure if s.email_notifications else None
    hooks = s.webhook_notifications.on_failure if s.webhook_notifications else None
    task_alerts = any(
        (t.email_notifications and t.email_notifications.on_failure)
        or (t.webhook_notifications and t.webhook_notifications.on_failure)
        for t in (s.tasks or [])
    )
    scheduled = bool(s.schedule or s.trigger or s.continuous)
    if scheduled and not (email or hooks or task_alerts):
        print(f"{base.job_id}\t{s.name}\tNO FAILURE ALERT")

Only scheduled, triggered or continuous jobs are reported. Manual one-off jobs are left out on purpose. On large workspaces this makes one API call per job, so run it off-peak.

Also look for alerts pointing at people who left: export the email recipients from the same loop and compare them with your Entra ID group membership.

#Common problems

#Prevent it next time

• Put notifications, health rules and timeout_seconds in your Databricks Asset Bundle templates so every new job starts with them.
• Add a CI check on bundle changes: no production job without an on_failure target. Compute policies control clusters, not job notifications, so they can't enforce this.
• Run the audit script weekly and send its output to the platform channel.
• Review thresholds every quarter. Data grows, and a warning set a year ago may now fire every night, which teaches people to ignore it.