Event-driven Databricks jobs with table update triggers: debounce settings, ALL vs ANY and incremental processing that survives retries

Gold jobs that run every 15 minutes "in case silver changed" waste compute when nothing changed and add latency when something did. Table update triggers start a job when one or more Unity Catalog tables get a new commit. This post covers the settings that matter at scale, the trade-off between the two trigger conditions, and how to write the triggered job so a duplicate or missed trigger doesn't corrupt anything.

#When to use which trigger

#Requirements and limits

• Monitored objects: Unity Catalog managed Delta and Iceberg tables, external Delta tables, materialized views, streaming tables, and views or metric views over supported tables. Views over read_files, over non-Unity Catalog tables or over federated tables are not supported.
• The job's run-as principal needs SELECT on every monitored object, plus USE CATALOG and USE SCHEMA. Without them the trigger fails to evaluate and the job just doesn't run. Nothing fails visibly unless you configured failure notifications.
• A trigger can monitor up to 25 tables. Above 10, every monitored table must have file events enabled on its external location. There are also workspace-wide caps on jobs and triggers; check the current limits before you roll this out to hundreds of jobs.

Warning

A revoked grant or a changed run-as identity silently stops a triggered job. Add an on-failure notification to every triggered job, and a freshness check on the downstream table (see the last section). Don't rely on the trigger alone.

#Configure the trigger

In the UI: Jobs & Pipelines > job > Schedules & Triggers > Add trigger > Table update, then pick the tables and the condition. For production, define it in a bundle so it's reviewed and versioned:

resources:
  jobs:
    gold_customer_360:
      name: gold-customer-360
      max_concurrent_runs: 1
      queue:
        enabled: true
      trigger:
        pause_status: UNPAUSED
        table_update:
          table_names:
            - prod.silver.customers
            - prod.silver.orders
          condition: ANY_UPDATED
          min_time_between_triggers_seconds: 600
          wait_after_last_change_seconds: 120
      tasks:
        - task_key: build
          notebook_task:
            notebook_path: ../src/build_customer_360.py
            base_parameters:
              updated_tables: "{{job.trigger.table_update.updated_tables}}"
      email_notifications:
        on_failure: [oncall-data@contoso.com]

Field names follow the Jobs API trigger.table_update object. If databricks bundle validate rejects a field, your CLI's schema is older than the feature, so upgrade the CLI.

databricks bundle validate -t prod
databricks bundle deploy -t prod
databricks jobs get <job-id> | jq '.settings.trigger'

#Tune the two timers

The two advanced settings decide how many runs you get and how stale the result can be:

• min_time_between_triggers_seconds is a cooldown after a run is triggered. Commits during the cooldown are collected and fire once it ends.
• wait_after_last_change_seconds is a settling window. Each new commit restarts it, so the job runs only after the table has been quiet for that long.

When both are set, the cooldown applies first and then the settling window.

Tip

A settling window on a continuously written table delays the job forever. For streaming sources, use only the cooldown. Its value is your acceptable staleness.

#ANY_UPDATED vs ALL_UPDATED

ANY_UPDATED fires when any monitored table commits. It's the right choice when the job processes whichever inputs changed.

ALL_UPDATED fires once every monitored table has had at least one update since the last run. It has no time window. If customers updates on Monday and orders on Thursday, the job runs on Thursday with Monday's customer change. If one upstream stops, the job never runs. Pair ALL_UPDATED with a freshness alert on each input.

#Make the triggered job idempotent

Triggers tell you that something changed, not exactly what you have already processed. Retries, repairs, manual runs, and maintenance commits such as OPTIMIZE (whether these fire the trigger has been reported both ways, so test on your workspace) all mean a run can't assume "one trigger = one new batch". Track the last processed Delta version per source in a control table and read the change data feed from there:

import json
from delta.tables import DeltaTable

sources = ["prod.silver.customers", "prod.silver.orders"]
ctl = "prod.ops.job_watermarks"
job = "gold-customer-360"

spark.sql(f"""CREATE TABLE IF NOT EXISTS {ctl}
              (job STRING, source STRING, last_version BIGINT, updated_at TIMESTAMP)""")

# Logged for audit only; processing is driven by the watermarks
print("trigger reported:", dbutils.widgets.get("updated_tables"))

for src in sources:
    row = spark.sql(f"SELECT last_version FROM {ctl} WHERE job='{job}' AND source='{src}'").first()
    start = (row.last_version + 1) if row else 0
    current = DeltaTable.forName(spark, src).history(1).select("version").first()[0]
    if start > current:
        continue  # nothing new: maintenance-only commit or duplicate trigger

    changes = (spark.read.option("readChangeFeed", "true")
               .option("startingVersion", start)
               .option("endingVersion", current)
               .table(src)
               .filter("_change_type IN ('insert','update_postimage','delete')"))

    apply_changes(src, changes)  # your MERGE into the gold table

    spark.sql(f"""MERGE INTO {ctl} t
                  USING (SELECT '{job}' job, '{src}' source, {current} v) s
                  ON t.job = s.job AND t.source = s.source
                  WHEN MATCHED THEN UPDATE SET last_version = s.v, updated_at = current_timestamp()
                  WHEN NOT MATCHED THEN INSERT VALUES (s.job, s.source, s.v, current_timestamp())""")

This requires delta.enableChangeDataFeed = true on each source. Three details matter:

• Update the watermark after the gold MERGE succeeds, and make the MERGE itself deterministic (deduplicate the changes by key and latest commit version), so a rerun of the same version range gives the same result.
• If the job falls behind longer than the source's log and file retention, the change feed for startingVersion is gone. Detect that error and fall back to a full rebuild rather than skipping data.
• max_concurrent_runs: 1 with queueing on stops two triggered runs from racing on the same watermark.

#Watch it

Triggered runs show up in system.lakeflow.job_run_timeline with trigger_type = 'TABLE'. A cheap freshness check finds jobs that should have run but didn't:

WITH last_run AS (
  SELECT job_id, max(period_end_time) AS last_end
  FROM system.lakeflow.job_run_timeline
  WHERE trigger_type = 'TABLE' AND period_start_time > now() - INTERVAL 7 DAYS
  GROUP BY job_id)
SELECT * FROM last_run
WHERE last_end < now() - INTERVAL 6 HOURS;

Also check DESCRIBE HISTORY on the inputs. If the inputs have commits after last_end and there's no run, look at grants and the run-as identity first.

#Prevent it next time

• Define triggers in bundles with explicit cooldowns. Don't leave the defaults on high-commit tables.
• Give every triggered job an on-failure notification and a downstream freshness alert.
• Drive processing from stored Delta versions, never from "the trigger fired, so there's one new batch".
• Enable file events on external locations before a trigger grows beyond 10 tables.
• Use a service principal as run-as so grants don't vanish when someone leaves.