Airflow deferred tasks never resume: triggerer down, blocked event loop, missing providers and capacity limits

#Symptoms

• Task instances show state deferred and never move to running, success or failed.
• Deferrable sensors (deferrable=True) wait far longer than the condition they watch.
• After an upgrade or image change, all deferred tasks stopped resuming.
• Triggerer logs show Triggerer's async thread was blocked for X seconds, likely by a badly-written trigger.

#How deferral works

1. A worker runs the task, which defers: it saves a trigger (a small async Python object) and exits, freeing the worker slot.
2. The triggerer process runs all triggers in an asyncio event loop.
3. When a trigger fires, the task is rescheduled and a worker resumes it.

So deferred tasks depend on a separate component, the triggerer. If it's missing, unhealthy or blocked, nothing resumes.

#Step 1: is a triggerer running?

# Airflow CLI: any live triggerer job?
airflow jobs check --job-type TriggererJob --allow-multiple --limit 5

On Kubernetes (official Helm chart):

kubectl -n airflow get pods -l component=triggerer
kubectl -n airflow logs deploy/airflow-triggerer --tail=200

Also check the UI health/status page: the triggerer heartbeat must be recent. If no triggerer was ever deployed (common after enabling deferrable operators on an older installation), deploy one (airflow triggerer, or triggerer.enabled: true in the Helm chart).

#Step 2: what is the triggerer doing?

-- Metadata DB: triggers and the task instances waiting on them
SELECT t.id, t.classpath, t.created_date, t.triggerer_id
FROM   trigger t
ORDER  BY t.created_date
LIMIT  50;

SELECT dag_id, task_id, run_id, state, trigger_id, trigger_timeout
FROM   task_instance
WHERE  state = 'deferred'
ORDER  BY trigger_timeout NULLS LAST
LIMIT  50;

Direct metadata DB queries are for diagnosis only; in Airflow 3, application code must not access the metadata DB, but administrators can still read it.

#Step 3: blocked event loop

All triggers in one triggerer share one event loop. A trigger that calls blocking code (synchronous HTTP, time.sleep, a blocking SDK) freezes every other trigger in that process. The log warns:

Triggerer's async thread was blocked for 12.34 seconds, likely by a badly-written trigger.

Fixes:

• For your own triggers: use async libraries (aiohttp, asyncio.sleep, async SDK clients), or wrap blocking calls with asyncio.to_thread(...) / sync_to_async.
• For provider triggers: upgrade the provider package; many early deferrable operators had blocking calls fixed in later releases.
• Find the culprit: the warning often includes the trigger; otherwise check which classpaths are active when the warnings appear.

# Good: async trigger polling an API
class JobDoneTrigger(BaseTrigger):
    async def run(self):
        async with aiohttp.ClientSession() as s:
            while True:
                async with s.get(self.status_url) as r:
                    if (await r.json())["state"] == "DONE":
                        yield TriggerEvent({"status": "done"})
                        return
                await asyncio.sleep(self.poll_interval)

#Step 4: missing or mismatched providers

The triggerer imports trigger classes by classpath. If its image lacks the provider package, or has a different version than the workers, triggers fail to load:

ModuleNotFoundError: No module named 'airflow.providers.databricks.triggers'

Make sure the scheduler, workers and triggerer all use the same image (or the same constraints file). This is the most common cause after upgrades where only worker images were rebuilt.

#Step 5: capacity

Each triggerer runs up to [triggerer] capacity triggers (named default_capacity in some versions; default 1000). Thousands of deferred sensors can exceed it.

airflow config get-value triggerer capacity 2>/dev/null || airflow config get-value triggerer default_capacity

Options:

• Run multiple triggerers (they share load and provide HA): triggerer.replicas: 2 in the Helm chart.
• Raise capacity if CPU and memory are low.
• Increase sensor poke_interval / trigger poll intervals; 10,000 triggers polling every 5 seconds is a lot of API calls.

#Step 6: timeouts and stuck tasks

Deferred tasks honour the operator's timeout (for sensors) or execution_timeout; when the trigger timeout passes, the task fails instead of waiting forever. Set them, so a lost trigger produces a visible failure.

To release tasks stuck after a broken triggerer was fixed, clear them so they run again:

airflow tasks clear my_dag -t wait_for_file -s 2026-10-27 -e 2026-10-28 --only-running --yes

(Options vary by version; airflow tasks clear --help.)

#Decision table

#Prevent it next time

• Treat the triggerer as a core component: health checks, at least two replicas, alerts on heartbeat.
• Build one image for all Airflow components.
• Review custom triggers for blocking calls in code review.
• Always set timeouts on deferrable sensors.