Home / Data Platform / Streaming & CDC

Data PlatformHow-toIntermediate

Advisory: batch, micro-batch or streaming - deciding how fresh data really needs to be

Real-time pipelines cost more to build and run, and many requests for "real time" don't need it. Turn freshness requests into explicit latency requirements, map them to batch, micro-batch or streaming, account for the operational cost of each, and use hybrid designs where only part of the data must be fast.

Deepak Bhardwaj3 min read

Applies toBatch ETL/ELT (dbt, Spark, ADF), incremental micro-batch (Spark Structured Streaming triggers, Auto Loader, Snowpipe), streaming (Kafka, Kinesis, Event Hubs, Flink, Spark streaming)

Turn "real time" into a number

Ask the business user:

Question Example answer
What decision or action uses this data? Block a fraudulent card payment
How quickly after the event must it happen? < 2 seconds
What happens if it's 5 minutes late? 1 hour? Fraud passes / no impact for reporting
How often is the data looked at? Dashboard checked twice a day

Most dashboards checked twice a day are well served by hourly or even daily data.

Map latency to approach

Required freshness Approach Examples
Hours–daily Batch Finance reporting, monthly KPIs
5–60 minutes Incremental micro-batch (scheduled) Operational dashboards, near-real-time BI
Seconds–1 minute Continuous micro-batch / streaming Inventory availability, monitoring
Sub-second, per-event action Event-driven streaming Fraud checks, personalisation, alerting

What each costs to run

Batch Micro-batch Streaming
Compute Runs then stops Frequent short runs or always-on small cluster Always on
Failure handling Re-run the job Checkpoints, idempotent writes Checkpoints, exactly-once/idempotency, replay
Late/out-of-order data Natural (process after cut-off) Lookback windows Watermarks, state management
On-call Morning check Business hours alerts 24×7 alerts on lag
Skills SQL/ELT Spark/streaming basics Stream processing, state, ops

Streaming also adds infrastructure (brokers, schema registry, connectors) that needs its own operations.

Hybrid designs

  • Hot path / cold path: stream only the few fields needed for the real-time use; full detail arrives by batch.
  • Streaming ingestion, batch transformation: land events continuously (Auto Loader/Snowpipe/Kafka sink) and transform on a schedule. Raw data is fresh and models are cheap.
  • Same code, different trigger: Spark Structured Streaming with availableNow trigger runs incrementally on a schedule today and can switch to continuous later.
(spark.readStream.table("bronze.orders")
   .writeStream.trigger(availableNow=True)          # micro-batch on a schedule
   .option("checkpointLocation", "/chk/silver_orders")
   .toTable("silver.orders"))

Decision record

For each pipeline: required freshness (with the business reason), chosen approach, SLA/SLO, alerting, owner, and the cost estimate. Revisit when the use changes.

Design review questions

  • What breaks if this data is an hour late?
  • Who gets paged if the stream stops at 3 AM, and is that justified?
  • How do we reprocess a day of data after a bug?
  • How does late or duplicate data get handled?

Tip

Start with incremental micro-batch on a schedule. It gives most of the freshness benefit, keeps operations simple and can move to continuous streaming later with little code change.

Checklist

  • Freshness requirement written as a number with a business reason
  • Approach mapped from latency table
  • Operating cost and on-call impact considered
  • Hybrid design considered
  • Reprocessing and late-data handling designed
  • Decision recorded per pipeline

Related fixes