Sev1: Auto Loader stream stuck or failing — checkpoint, schema and notification problems

#Symptoms

• New files land in storage, but the bronze table doesn't grow.
• The stream fails right after start with UnknownFieldException or a checkpoint error.
• The stream runs but takes hours per batch (huge directory listing).

#1. Read the error and progress

q = [s for s in spark.streams.active if s.name == "bronze_orders"]
print(q[0].lastProgress if q else "not running")

In a job, look at the failed run's error. Then match:

#2. Schema evolution stop (expected behaviour)

With cloudFiles.schemaEvolutionMode = addNewColumns (default when a schema location is used), Auto Loader stops the stream once when it detects new columns, after updating the schema. Restarting picks up the new schema.

Fix: make the job retry automatically (job task retries ≥ 1), or choose another mode:

.option("cloudFiles.schemaEvolutionMode", "rescue")   # keep schema; new fields go to _rescued_data

#3. Checkpoint problems

The checkpoint records which files were processed. Rules:

• One checkpoint per stream; never share or copy between streams.
• Don't move the checkpoint or change the source path for an existing checkpoint.

If the checkpoint is lost or corrupted, start a new checkpoint but avoid reloading everything:

(spark.readStream.format("cloudFiles")
   .option("cloudFiles.format", "json")
   .option("cloudFiles.schemaLocation", "/Volumes/main/ingest/_schemas/orders_v2")
   .option("modifiedAfter", "2026-10-24T00:00:00.000000Z")    # only files after the last good load
   .load("abfss://landing@stdata.dfs.core.windows.net/orders/")
 .writeStream
   .option("checkpointLocation", "/Volumes/main/ingest/_chk/orders_v2")
   .toTable("main.bronze.orders"))

Or use cloudFiles.includeExistingFiles = false to process only new arrivals. Then deduplicate any overlap in silver (MERGE on a business key or on _metadata.file_path).

Warning

A new checkpoint with default settings reprocesses every existing file in the path. On a large landing zone that's both slow and a duplicate-data incident.

#4. Nothing picked up

• Path/glob: compare the load path with where files actually land (case, date folders).
• File notification mode: check the Event Grid subscription and storage queue Databricks created:

az eventgrid event-subscription list --source-resource-id <storage-account-id> -o table
az storage queue list --account-name stdata --auth-mode login -o table

The service principal/credential used for notifications needs rights to create and read these. If someone deleted the subscription, the stream sees no new files. Recent Databricks releases also offer managed file events on Unity Catalog external locations, which avoid per-stream setup.

• Same file names overwritten: Auto Loader processes a file path once by default; overwritten files with the same name are ignored unless cloudFiles.allowOverwrites is set.

#5. Slow listing

Directory listing mode lists the whole path each batch. For millions of files, switch to file notification mode / file events, and archive processed files (cloudFiles.cleanSource where available) or partition the landing path by date.

#Prevent it next time

• Job retries on, and alerts on stream failure and on bronze table freshness.
• Checkpoints and schema locations in a dedicated UC volume nobody cleans up by hand.
• File notifications or file events for high-volume paths.