Databricks operations monitoring: system tables, alerts and an ops dashboard

#1. Enable the system schemas

An account admin enables the system schemas you need (access, billing, compute, lakeflow, query, ...) per metastore, and grants read access to the platform team group:

GRANT USE CATALOG ON CATALOG system TO `grp-platform-ops`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.lakeflow TO `grp-platform-ops`;

#2. Job health

-- failed job runs in the last 24 hours
SELECT t.workspace_id, t.job_id, j.name, t.run_id, t.result_state, t.period_end_time
FROM system.lakeflow.job_run_timeline t
LEFT JOIN system.lakeflow.jobs j USING (workspace_id, job_id)
WHERE t.period_end_time >= now() - INTERVAL 24 HOURS
  AND t.result_state IN ('FAILED', 'TIMED_OUT', 'ERROR')
ORDER BY t.period_end_time DESC;

-- runs taking 2x longer than their 30-day median
WITH d AS (
  SELECT job_id, run_id, (unix_timestamp(period_end_time) - unix_timestamp(period_start_time)) AS secs
  FROM system.lakeflow.job_run_timeline WHERE period_start_time >= current_date() - 30 AND result_state = 'SUCCEEDED')
SELECT d.job_id, d.run_id, d.secs, m.median_secs
FROM d JOIN (SELECT job_id, percentile(secs, 0.5) AS median_secs FROM d GROUP BY job_id) m USING (job_id)
WHERE d.secs > 2 * m.median_secs;

Also set job-level notifications (email, Teams/Slack webhooks) and duration warning thresholds for critical jobs.

#3. Data freshness (the SLA users care about)

SELECT 'orders' AS tbl, MAX(ingest_ts) AS last_load,
       timestampdiff(MINUTE, MAX(ingest_ts), now()) AS minutes_stale
FROM prod_sales.orders.orders;

Create a SQL alert per critical table when minutes_stale > the SLA. Lakehouse Monitoring adds data quality and drift profiles for important tables.

#4. Compute

-- clusters running without auto-termination or policy (cost and risk)
SELECT cluster_id, cluster_name, owned_by, auto_termination_minutes, policy_id
FROM system.compute.clusters
WHERE delete_time IS NULL AND (auto_termination_minutes IS NULL OR auto_termination_minutes = 0 OR policy_id IS NULL);

For start failures and unexpected terminations, read cluster events (API/CLI) for the clusters behind failed job runs:

databricks clusters events 0123-456789-abcde123 --output json | jq ".events[] | {timestamp, type, details}"

Watch for capacity errors (cloud quota, spot unavailability) — see the cluster start failure post.

#5. Warehouses

Queueing (waiting_at_capacity_duration_ms in system.query.history) and failed statements per warehouse.

#6. Security from the audit log

SELECT event_time, user_identity.email, service_name, action_name, request_params, response.status_code
FROM system.access.audit
WHERE event_date >= current_date() - 1
  AND (action_name IN ('createToken','changeWorkspaceAcl','updatePermissions','deleteTable')
       OR response.status_code = 403)
ORDER BY event_time DESC;

Alert on admin role changes, token creation by service accounts you don't expect, mass deletions, and IP access list changes. Also send diagnostic logs to your SIEM (Azure Monitor diagnostic settings / S3 log delivery).

#7. Route alerts

#8. One ops dashboard

AI/BI dashboard with: failures (24h), SLA misses, top cost jobs, queueing, freshness of critical tables, security events. Review it in the daily stand-up.

Tip

Alert on freshness, not only on failures. A job that "succeeds" but loads zero rows, or a schedule that was paused by mistake, only shows up as stale data.