Applies toSpark Structured Streaming and Auto Loader on Azure Databricks (job compute or serverless), sources such as Event Hubs, Kafka and cloud files
Inventory streams first
| Question | Why |
|---|---|
| What latency does the consumer actually need? | Many "streams" serve hourly dashboards |
| How busy is the stream? | Mostly idle streams waste an always-on cluster |
| How big is the state? | Drives memory and VM choice |
| How many streams per cluster? | One cluster per tiny stream is expensive |
Options by latency need
| Need | Approach |
|---|---|
| Minutes to hours | trigger(availableNow=True) run by a scheduled job; cluster stops between runs |
| Tens of seconds to minutes | Continuous job with processingTime trigger (e.g. 1–5 min) |
| Seconds | Continuous with default/short trigger, sized for peak |
(spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.load("abfss://landing@stlake.dfs.core.windows.net/orders/")
.writeStream
.trigger(availableNow=True) # process what's there, then stop
.option("checkpointLocation", "abfss://chk@stlake.dfs.core.windows.net/orders/")
.toTable("bronze.orders"))
Tune continuous streams
| Setting | Effect |
|---|---|
trigger(processingTime="2 minutes") | Fewer, larger micro-batches; fewer small files and commits |
maxOffsetsPerTrigger (Kafka/Event Hubs Kafka) / maxEventsPerTrigger | Caps batch size so a backlog doesn't need a huge cluster |
cloudFiles.maxFilesPerTrigger / maxBytesPerTrigger | Same for Auto Loader |
| Shuffle partitions | Match to cluster cores for small streams (default 200 is often too many) |
State
- Use the RocksDB state store for large state:
spark.conf.set("spark.sql.streaming.stateStore.providerClass", "com.databricks.sql.streaming.state.RocksDBStateStoreProvider") - Set watermarks so old state is dropped.
- Watch state size in the streaming query progress metrics.
Pack light streams
Several low-volume streams can share one job cluster (multiple streaming tasks or one notebook starting several queries). Give each its own checkpoint and, if needed, a scheduler pool:
spark.sparkContext.setLocalProperty("spark.scheduler.pool", "orders")
Keep heavy or critical streams on their own compute so one can't starve another.
Measure
StreamingQueryProgress: input rows/sec, processing rate, batch duration.- Cost per stream from system billing tables by job ID.
- Lag at the source (Event Hubs consumer lag; see the Event Hubs lag Sev1 post).
Tip
Moving a stream from always-on to
availableNowevery 15 minutes often cuts its compute cost sharply with little change to how fresh the data feels to users. Agree the latency with the consumer first.
Checklist
- Every stream's latency need confirmed
- availableNow used where minutes are acceptable
- Trigger intervals and batch caps tuned
- RocksDB and watermarks for large state
- Light streams packed, heavy ones isolated
- Cost per stream tracked