First 15 minutes
- Confirm what's missing: which tables, which dates/hours.
- Tell consumers: "Data for 24 Oct is delayed; ETA to follow". A clear message stops ten parallel investigations.
- Find the stall point.
-- Which partitions arrived?
SELECT load_date, COUNT(*) FROM silver.sales WHERE load_date >= current_date() - 3 GROUP BY load_date ORDER BY load_date;
1. Where did the chain stop?
airflow dags list-runs -d sales_daily --start-date 2026-10-23 -o table
airflow tasks states-for-dag-run sales_daily scheduled__2026-10-24T02:00:00+00:00
| Task state | Typical cause |
|---|---|
upstream_failed | An earlier task failed — fix that first |
Sensor up_for_reschedule for hours | Upstream file/table never arrived |
queued forever | Pool or executor capacity exhausted |
success but no data | Task succeeded on an empty input — add row-count checks |
Check whether the source delivered at all (file landing zone, source system extract status). If the source is late, the backfill waits for it.
2. Fix the cause, then clear and re-run
For a single failed run, clear from the failed task — downstream tasks re-run automatically:
airflow tasks clear sales_daily -t "transform_sales" -s 2026-10-24 -e 2026-10-24 --downstream --yes
3. Backfill multiple intervals
airflow dags backfill sales_daily -s 2026-10-21 -e 2026-10-24 --reset-dagruns --max-active-runs 1
In Airflow 3, backfills are created through the API/UI (airflow backfill create) and run by the scheduler; check your version's command.
--max-active-runs 1runs intervals in order, which matters when each day depends on the previous one (snapshots, SCD2, running totals).- Use pools to cap load on source databases.
Warning
Backfills re-run every task in each interval. If tasks append instead of overwrite/MERGE, you'll create duplicates. Check idempotency before you start.
4. Mark what shouldn't run
If some intervals don't need processing (source holiday, intentionally skipped), mark them instead of running them. In the UI: open the DAG run → Mark success. From the CLI, mark a single task successful for that interval:
airflow tasks run sales_daily load_sales 2026-10-22 --mark-success
Record why in the run note so the next person doesn't backfill it again.
5. Close the loop
- Validate row counts and key totals for each backfilled interval against the source.
- Refresh downstream semantic models and extracts.
- Send the "data restored" message with the time and any caveats.
Prevent it next time
- SLAs/deadline alerts on the tasks that produce the data consumers need (Airflow SLA miss callbacks or external freshness checks on the tables).
- Sensors with timeouts and
mode="reschedule"so they don't hold worker slots for hours. - Row-count and freshness checks that fail the run instead of succeeding on empty input.