dbt microbatch incremental models in production - event_time filtering, lookback, backfills and failed batches

#Goal

You have a large event-based fact model built with a hand-written is_incremental() filter. It misses late rows, backfills mean ad hoc --vars hacks, and one failed run forces a rebuild of the whole window. The microbatch strategy (dbt 1.9+) changes how this works. dbt splits the run into time batches, filters each upstream input for you, writes each batch with an idempotent replace, and records which batches failed.

This post covers the parts that trip experienced teams up: which inputs actually get filtered, how each adapter writes a batch, lookback cost, backfill commands and what --full-refresh does.

#How a microbatch run works

For each batch, dbt renders your model SQL once with a time window [event_time_start, event_time_end). Inside that render, every ref() or source() whose node has event_time configured gets a filter on that window. Then the adapter replaces that window in the target:

Because each batch replaces its own window, re-running a batch is idempotent. That property is what makes backfills and retries safe. The model must produce only rows that belong to the current batch's window, or a later batch will delete them.

#Configure the model

-- models/marts/fct_page_views.sql
{{ config(
    materialized='incremental',
    incremental_strategy='microbatch',
    event_time='viewed_at',
    begin='2024-01-01',
    batch_size='day',
    lookback=2,
    full_refresh=false
) }}

select
    v.view_id,
    v.viewed_at,
    v.user_id,
    u.country,
    v.page_path
from {{ ref('stg_page_views') }} v
left join {{ ref('dim_users') }} u on u.user_id = v.user_id

• event_time on the model is the column dbt uses to assign output rows to batches and to delete or overwrite the target window.
• batch_size is hour, day, month or year. Pick the grain at which late data actually lands. Hourly batches on a table that gets corrections days later just multiply queries.
• lookback (default 1) reprocesses that many batches before the latest bookmark on every normal run. lookback=2 with daily batches means every run rebuilds the last two days plus today.
• begin is where an initial build starts. dbt doesn't look up the earliest date in your data.

Warning

dbt treats event_time, begin and the backfill flags as UTC. If viewed_at is stored in local time without a zone, batch boundaries are shifted by the offset and rows near midnight land in the "wrong" batch. Convert to UTC in staging.

#Make sure the parents are filtered

This is the most common mistake. Filtering only applies to inputs that have event_time configured. A parent without it is read in full for every batch. A 365-batch backfill then scans the whole upstream table 365 times.

# models/staging/_staging.yml
models:
  - name: stg_page_views
    config:
      event_time: viewed_at

sources:
  - name: web
    tables:
      - name: raw_page_views
        config:
          event_time: received_at

Dimensions are the exception: dim_users needs the full table in each batch, because a view on Tuesday joins users created years ago. Leave event_time off it, or opt out explicitly so readers understand it's deliberate:

left join {{ ref('dim_users').render() }} u on u.user_id = v.user_id

.render() returns the plain relation without the batch filter. Use it for small or slowly changing inputs only.

Check what dbt actually ran before trusting a model in production:

dbt compile -s fct_page_views
# inspect target/run/... after a run for the per-batch SQL
dbt run -s fct_page_views --event-time-start "2026-11-01" --event-time-end "2026-11-02" --debug

Each batch's query should have >= start and < end predicates on every event-time parent.

#When the source time and the model time differ

Filtering is on each parent's own event_time. If raw_page_views.received_at can lag viewed_at by hours, a batch filtered on received_at for 2026-11-05 includes events with viewed_at on 2026-11-04. Those rows are written into the 2026-11-05 window of the target, and then the 2026-11-04 replace can't see them. Configure the parent's event_time on the same business timestamp the model uses, and handle arrival lag with lookback.

#Backfills

Rebuild a specific range without touching the rest:

dbt run -s fct_page_views \
  --event-time-start "2026-10-01" --event-time-end "2026-10-08"

Both flags are required together. The range is split into batches, and each one is replaced independently. Run backfills downstream too, with a selector:

dbt run -s fct_page_views+ \
  --event-time-start "2026-10-01" --event-time-end "2026-10-08"

Downstream microbatch models rebuild the same window. Non-microbatch incremental models downstream ignore the flags and run their normal logic, so check that their own filters can pick up the corrected rows.

#Failed batches and retry

If 3 of 30 batches fail (a warehouse timeout, a lock), the other 27 are already committed. The node is reported as a partial success. Don't rerun the whole range:

dbt retry

dbt retry reads the previous run's artifacts and reprocesses only the failed batches. Keep target/run_results.json from the failed run, so in CI persist target/ between the failed step and the retry step.

#Parallel batches

dbt can run batches concurrently when the adapter supports it and the model is safe to parallelise. For example, a model that references {{ this }} is treated as order-dependent. Override with concurrent_batches:

{{ config(concurrent_batches=false) }}   -- cumulative logic, running totals

Parallelism is bounded by threads. On Snowflake, a large backfill with high threads can queue on a small warehouse. Run backfills with a dedicated target or warehouse.

#Full refresh

{{ config(full_refresh=false) }}

With full_refresh=false, a stray dbt build --full-refresh in a deploy job won't rebuild years of history. A real full rebuild then becomes a deliberate, bounded command:

dbt run -s fct_page_views --full-refresh \
  --event-time-start "2024-01-01" --event-time-end "2026-11-09"

Caution

Without full_refresh=false, --full-refresh drops and rebuilds the table from begin. If the source retention is shorter than the history in the target (raw events kept 90 days, fact kept 3 years), the history before the retention window is lost. Check source retention before any full refresh.

#Common failure modes

#Prevent it next time

• Treat event_time as part of the contract for every staging model of an event table.
• Set full_refresh=false on every microbatch model.
• Add a data test that compares daily row counts between the model and its main parent for the last lookback + 1 days.
• Run backfills from a runbook with explicit --event-time-start/--event-time-end, a dedicated warehouse and dbt retry on failure.