Airflow ExternalTaskSensor waits forever: logical date mismatch, execution_delta and failed upstream states

#Symptoms

• A task built on ExternalTaskSensor sits in running or up_for_reschedule for hours.
• The upstream DAG shows a green run for "today" in the UI.
• The sensor log repeats a line like Poking for tasks ['load_orders'] in dag orders_ingest on 2026-11-08T02:00:00+00:00 ... until it times out, or forever.
• Sometimes the opposite: the upstream task failed, and the sensor keeps waiting instead of failing.

The sensor looks for an upstream task instance (or DAG run) with a specific logical date. If that exact run doesn't exist, it waits. In most cases nothing is wrong with the upstream DAG. The sensor is looking for the wrong date.

#How the sensor decides

With no execution_delta, the sensor looks for an upstream run with exactly the same logical date as its own run. Two DAGs on different schedules never share a logical date, so the sensor never succeeds.

#Check 1: compare the logical dates

Get the logical date the sensor is waiting for from its log, then list the upstream runs:

# Airflow 2 and 3
airflow dags list-runs orders_ingest --start-date 2026-11-07 --end-date 2026-11-09
# Airflow 2.x older builds use: airflow dags list-runs -d orders_ingest

Check the logical_date column (named execution_date in Airflow 2 output). Then check the task state for that run:

airflow tasks states-for-dag-run orders_ingest <run_id>

Typical findings:

#Check 2: fix the date offset

If the upstream runs one hour earlier, tell the sensor:

from datetime import timedelta

# Airflow 3: from airflow.providers.standard.sensors.external_task import ExternalTaskSensor
# Airflow 2: from airflow.sensors.external_task import ExternalTaskSensor
from airflow.providers.standard.sensors.external_task import ExternalTaskSensor

wait_orders = ExternalTaskSensor(
    task_id="wait_orders_ingest",
    external_dag_id="orders_ingest",
    external_task_id="load_orders",
    execution_delta=timedelta(hours=1),   # my 03:00 run waits for the 02:00 upstream run
    allowed_states=["success"],
    failed_states=["failed", "upstream_failed"],
    check_existence=True,
    mode="reschedule",
    poke_interval=300,
    timeout=4 * 60 * 60,
)

execution_delta is subtracted from the sensor's logical date. A positive delta looks backwards. A common mistake is using the difference in the wrong direction.

Tip

Remember that in Airflow 2, a scheduled run's logical date is the start of the data interval, not the time it ran. A @daily DAG that runs at 00:00 on 9 November has logical date 8 November. Compare logical dates, not "ran at" times, and the offset becomes obvious.

For irregular schedules, a function can return the matching date. Recent versions of the standard provider deprecate execution_date_fn and accept only one of execution_delta, execution_date_fn or the newer date parameter. Parameter names vary by version, so check your provider's documentation before you use it.

#Check 3: make failures fail fast

By default failed_states is empty. If the upstream task fails, the sensor keeps waiting until timeout, and with no timeout set it waits for the default (7 days for sensors). Set:

• failed_states=["failed", "upstream_failed"] so an upstream failure fails the sensor immediately.
• Optionally skipped_states=["skipped"] (where your version supports it) if a skipped upstream should skip this task.
• check_existence=True so a renamed DAG or task fails at once instead of producing a silent wait.

Warning

Adding "skipped" to allowed_states makes the downstream run on a day the upstream chose not to load. Only do this if an empty load is a valid input for the downstream DAG.

#Check 4: the sensor is holding a worker slot

A sensor in poke mode (the default) holds a worker slot for the whole wait. Ten sensors waiting six hours can starve the pool and make other tasks queue. Use:

• mode="reschedule": frees the slot between pokes, shows up_for_reschedule.
• deferrable=True: hands the wait to the triggerer. A triggerer must be running.

Check how many slots sensors are using:

airflow pools list
airflow tasks states-for-dag-run reporting_daily <run_id>

#Check 5: manual runs and backfills

A manually triggered downstream run gets a logical date equal to the trigger time (or whatever you entered). No upstream run has that date, so the sensor waits.

Options for a one-off:

# Trigger with the logical date the upstream actually has (Airflow 2.x syntax; check your version's flags)
airflow dags trigger reporting_daily --exec-date 2026-11-08T03:00:00+00:00

# After fixing execution_delta, clear the sensor so it re-checks with the new date
airflow tasks clear reporting_daily -t wait_orders_ingest -s 2026-11-08 -e 2026-11-08 --downstream

In Airflow 3 the trigger flag is --logical-date. Some Airflow 3 builds allow runs with no logical date at all. A sensor can't match those runs, so give manual runs a logical date. To mark a task success without running it, use the UI (task instance > Mark as > Success). It's the most reliable cross-version method.

#Symptom → cause → fix

#Prevent it next time

• Always set timeout, failed_states and check_existence on cross-DAG sensors.
• Use mode="reschedule" or deferrable=True for any wait longer than a few minutes.
• Put the upstream DAG's schedule in a comment next to execution_delta, so the next schedule change updates both.
• For DAG-to-DAG dependencies inside one Airflow, prefer datasets (Airflow 2.4+) or assets (Airflow 3). The downstream DAG starts when the upstream task updates the data, with no date matching at all:

from airflow.sdk import Asset   # Airflow 3; Airflow 2.4+: from airflow.datasets import Dataset

orders = Asset("s3://lake/orders/")
# upstream task: outlets=[orders]
# downstream DAG: schedule=[orders]

• Alert on sensors that run longer than usual. A timeout failure is a page you can act on. A sensor still poking at 09:00 is a missed SLA nobody saw coming.