Home / Databricks on Azure

Databricks on AzureHow-toIntermediate

Optimization: tuning Auto Loader on Azure - file notification vs listing, batch sizes and schema inference cost

Auto Loader works out of the box, but large landing zones make directory listing slow and costly. Switch to file notification or managed file events for big folders, cap batch sizes, avoid repeated schema inference, and keep landing paths well partitioned so discovery stays fast.

Deepak Bhardwaj2 min read

Applies toAuto Loader (cloudFiles) on Azure Databricks with ADLS Gen2, Event Grid and storage queues; Unity Catalog external locations with file events

Discovery modes

Mode How it finds files Fits
Directory listing (default) Lists the input path each micro-batch (incremental listing where possible) Small to medium folders, simple setup
File notification Event Grid → storage queue; Auto Loader reads queue messages Large folders, millions of files, frequent runs
Managed file events (Unity Catalog external locations) Databricks manages events for the external location Many streams on the same location; check availability

Signs listing has become the bottleneck: long time before each batch starts, high storage list transactions, runs that take longer as the folder grows.

File notification setup

(spark.readStream.format("cloudFiles")
   .option("cloudFiles.format", "json")
   .option("cloudFiles.useNotifications", "true")
   .option("cloudFiles.resourceGroup", "rg-data-prod")
   .option("cloudFiles.subscriptionId", "<subscription-id>")
   # identity with rights to create Event Grid subscriptions and queues
   .option("cloudFiles.clientId", dbutils.secrets.get("kv", "al-client-id"))
   .option("cloudFiles.clientSecret", dbutils.secrets.get("kv", "al-client-secret"))
   .option("cloudFiles.tenantId", "<tenant-id>")
   .load("abfss://landing@stlake.dfs.core.windows.net/orders/"))

Or create the Event Grid subscription and queue yourself and pass cloudFiles.queueName, which needs fewer permissions. Newer setups can use managed file events (cloudFiles.useManagedFileEvents) on Unity Catalog external locations; check current docs.

Batch sizing

Option Use
cloudFiles.maxFilesPerTrigger Cap files per micro-batch (default 1000)
cloudFiles.maxBytesPerTrigger Cap bytes per micro-batch
trigger(availableNow=True) Process the backlog in sized batches and stop

Caps keep a backlog from creating one giant batch that needs a huge cluster.

Schema inference cost

  • Inference samples files at stream start; with cloudFiles.schemaLocation the schema is stored and reused.
  • Use cloudFiles.schemaHints for known columns rather than inferring everything.
  • For stable feeds, provide the full schema and use cloudFiles.schemaEvolutionMode as needed (see the Auto Loader Sev1 post).

Landing zone layout

  • Partition landing paths by date (/orders/2026/10/25/) so listing and backfills stay bounded.
  • Move or lifecycle-delete processed files; Auto Loader tracks what it has seen in the checkpoint, not by deleting files.
  • cloudFiles.cleanSource can archive or delete processed files on newer runtimes; check support before relying on it.

Tip

Before switching modes, measure: time from batch start to first task, and storage list operations per hour for the landing account. That shows whether discovery is the problem.

Checklist

  • Discovery mode chosen by folder size and frequency
  • Notification permissions or pre-created queue in place
  • Batch caps set
  • Schema location and hints used
  • Landing paths partitioned and cleaned up
  • Before/after discovery time measured

Related fixes