Databricks file arrival trigger not firing: locations, permissions, file events and timing settings

#Symptom

You configured a job with a file arrival trigger on an ADLS path. Files land, but no run starts. Or runs start much later than expected, or several times for one batch of files.

#How the trigger works

• The trigger watches a Unity Catalog volume path or a path under a UC external location.
• Databricks checks the location for new files on a schedule (about every minute), or uses file events if they are enabled on the external location.
• When new files are found, it starts a run, subject to two settings:
  • wait_after_last_change_seconds: wait until no new files arrived for this long (lets a batch finish).
  • min_time_between_triggers_seconds: minimum gap between runs.
• Files that existed before the trigger was enabled don't count as new.

#Step 1: check the trigger configuration

databricks jobs get <job-id> --output json | jq '.settings.trigger'

Expected shape:

{
  "pause_status": "UNPAUSED",
  "file_arrival": {
    "url": "abfss://landing@stlakeprod01.dfs.core.windows.net/sales/orders/",
    "min_time_between_triggers_seconds": 60,
    "wait_after_last_change_seconds": 120
  }
}

• pause_status: PAUSED is the most common "bug". Jobs deployed from a bundle in development mode are often paused.
• The URL must end with / and point to a directory, and it must be inside a UC external location or be a volume path like /Volumes/prod/landing/orders/.
• Files in subdirectories of the URL count as arrivals. Files outside it don't, so check the producer writes where you think.

#Step 2: check the location and permissions

-- Which external location covers the path?
DESCRIBE EXTERNAL LOCATION landing_sales;

-- The job's run-as identity needs READ FILES on the location (or READ VOLUME on the volume)
SHOW GRANTS ON EXTERNAL LOCATION landing_sales;

The check runs as the job's run-as identity (owner or service principal). If that identity can't list the path, the trigger silently finds nothing. The trigger's status in the job UI usually shows a permission or path error; look there first.

#Step 3: file events and large directories

Without file events, the trigger lists the directory. Very large directories (many thousands of files) are slow to list and have documented limits on how many files a trigger can watch. Enable file events on the external location so Databricks uses storage notifications instead of listing:

• Catalog Explorer > External locations > your location > enable file events.
• Databricks creates or uses Azure Event Grid and a storage queue for the account. The access connector's managed identity needs the roles described in the setup page (for example on the storage account and resource group).

Check the current limits for listing-based triggers and for the number of file-arrival jobs per workspace in the Databricks documentation; they have changed over time.

Tip

Keep landing directories tidy: move processed files to an archive path or use date partitions, so the watched path doesn't grow forever.

#Step 4: networking

If the storage account has a firewall or private endpoints only, the trigger's file check (a Databricks control-plane / serverless service, not your cluster) must be allowed:

• Allow the workspace's access connector as a resource instance rule on the storage account, and make sure "trusted services" settings match Databricks guidance.
• For serverless and managed services, configure the network connectivity configuration (NCC) private endpoints or firewall rules for the storage account. See serverless storage firewall and NCC 403.

#Step 5: timing problems

#Make the job itself idempotent

A trigger tells you something arrived, not which files. Process new files with Auto Loader (cloudFiles) and a checkpoint, so each run picks up everything new exactly once, whether one run or three were triggered.

(spark.readStream.format("cloudFiles")
   .option("cloudFiles.format", "json")
   .load("/Volumes/prod/landing/orders/")
   .writeStream.option("checkpointLocation", "/Volumes/prod/chk/orders_bronze")
   .trigger(availableNow=True)
   .toTable("prod.bronze.orders"))

#Prevent it next time

• Set pause_status: UNPAUSED explicitly in production bundle targets.
• Alert if a file-triggered job hasn't run in longer than expected (system tables: system.lakeflow.job_run_timeline).
• Related: Auto Loader file notification tuning.