Airflow started hundreds of runs after an unpause or deploy: stop a catchup flood safely

#Symptom

Right after someone unpaused a DAG, renamed it or changed its start_date, Airflow fills the grid with runs for past dates. Source systems get hammered, warehouse costs jump, pools fill up and other DAGs sit waiting.

That's catchup. With catchup=True, the scheduler creates a run for every schedule interval that hasn't run yet, going back to the last run or, if there isn't one, to start_date.

#Step 1: stop new runs

Pause the DAG first:

airflow dags pause sales_daily

Pausing stops the scheduler from creating new runs and from scheduling new tasks for that DAG. Tasks that are already running will carry on.

Then look at the size of the problem. In Airflow 3:

airflow dags list-runs sales_daily --state queued -o plain | wc -l
airflow dags list-runs sales_daily --state running -o plain

In Airflow 2, pass the DAG with -d sales_daily. Check airflow dags list-runs --help on your version.

#Step 2: decide what to do with the runs

Don't delete anything yet. Ask the data owner a question: do you actually want the missed intervals?

#Step 3: cancel the queued runs

In the UI, filter the DAG's runs by state and mark the selected runs as failed. That's the safest method for a small number.

For larger numbers, use the REST API. Airflow 3 uses /api/v2, with a bearer token. Airflow 2 uses /api/v1, with whatever auth backend you've configured.

# Airflow 3
BASE=https://airflow.example.com/api/v2
TOKEN=<token>

curl -s -H "Authorization: Bearer $TOKEN" \
  "$BASE/dags/sales_daily/dagRuns?state=queued&limit=100" \
  | jq -r '.dag_runs[].dag_run_id' > queued-runs.txt

wc -l queued-runs.txt
head queued-runs.txt

Review the list, and then mark each run as failed:

while read -r run_id; do
  enc=$(jq -rn --arg v "$run_id" '$v|@uri')
  curl -s -o /dev/null -w "%{http_code} $run_id\n" -X PATCH \
    -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    -d '{"state":"failed"}' \
    "$BASE/dags/sales_daily/dagRuns/$enc"
done < queued-runs.txt

Run IDs contain + and :, hence the encoding. The page size limit applies, so repeat until the list is empty.

For runs that are in progress, check whether the running task can be interrupted safely. Marking a running task as failed asks the worker to stop it, but a query already sent to a warehouse may keep running. See stopping a runaway query if you're on Snowflake.

Caution

airflow dags delete removes the DAG's run history and metadata. It looks like the fast way out, but you lose your record of what ran. It also doesn't stop catchup. If the DAG file's still there and the DAG is unpaused with catchup=True, the scheduler may see no history and start from start_date again.

Don't modify the metadata database by hand either. Airflow 3 restricts direct access for good reason. See metadata DB access.

#Step 4: find the cause

airflow dags details sales_daily
airflow config get-value scheduler catchup_by_default

Look in Git for what changed around the time it started:

git log --since="3 days ago" -p -- dags/sales_daily.py

#Step 5: fix the DAG

Set this explicitly on every DAG, rather than relying on defaults:

from datetime import datetime
from airflow import DAG

with DAG(
    dag_id="sales_daily",
    schedule="0 2 * * *",
    start_date=datetime(2026, 11, 1),
    catchup=False,
    max_active_runs=1,
) as dag:
    ...

Import paths differ slightly between versions. Airflow 3 prefers airflow.sdk, while the older path has a compatibility layer. Check your version's documentation.

Before unpausing, check that it's been parsed, and then look at what will happen:

airflow dags list-import-errors
airflow dags details sales_daily | grep -iE "catchup|start_date|max_active"
airflow dags unpause sales_daily

Even with catchup=False, unpausing will create the most recent interval's run. Make sure that's what you want.

#Step 6: repair partial loads

A flood can leave the data in a mixed state. Check it instead of assuming:

SELECT load_date, COUNT(*) AS row_count, COUNT(DISTINCT order_id) AS distinct_orders
FROM sales.daily_orders
WHERE load_date >= '2026-10-01'
GROUP BY load_date
ORDER BY load_date;

Look for missing dates, rows counted twice and dates loaded with today's data rather than the right interval's. That last one happens when a task uses the current time instead of data_interval_start.

# Good: same result whenever it runs
sql = "SELECT * FROM src WHERE order_date = '{{ ds }}'"

Fix duplicates with the steps in duplicates after a rerun.

#Step 7: backfill properly if you need the dates

If the owner does want the history, run it in a controlled way. Airflow 3 has a backfill command:

airflow backfill create --dag-id sales_daily \
  --from-date 2026-10-01 --to-date 2026-10-31 \
  --max-active-runs 2

On Airflow 2, the older equivalent is airflow dags backfill -s 2026-10-01 -e 2026-10-31 sales_daily. Flags vary across releases, so check --help. Run it outside peak hours and use a pool to protect the source. For the wider process, see SLA breach backfill.

#Prevent it next time

• Make catchup and max_active_runs required in a DAG validation test in CI.
• Treat a dag_id or start_date change as a risky change in code review.
• Write tasks to be idempotent, so that a repeat run replaces the data instead of adding to it.
• Use pools for shared systems, so one DAG can't starve the rest.
• Alert when one DAG has more than a set number of queued runs. Also see tasks stuck in queued.