Symptoms
- Many task instances in
queuedorscheduledfor a long time; DAG runs don't progress. - Scheduler heartbeat is healthy, the UI works.
- Often after a worker deploy, a broker restart, or a large backfill.
1. Count where tasks are stuck
-- Against the Airflow metadata database (read-only)
SELECT state, pool, queue, COUNT(*)
FROM task_instance
WHERE state IN ('queued','scheduled')
GROUP BY state, pool, queue
ORDER BY 4 DESC;
| Pattern | Likely cause |
|---|---|
Mostly scheduled | Concurrency limit or pool full; scheduler won't queue more |
Mostly queued (Celery) | No worker consuming that queue, or broker problem |
Mostly queued (Kubernetes) | Pods can't be created (quota, image, node capacity) |
| One pool dominates | Pool slots exhausted |
2. Celery: are workers consuming?
airflow celery flower # or check your Flower UI
celery --app airflow.providers.celery.executors.celery_executor.app inspect active_queues
(The Celery app path differs between Airflow versions; older versions use airflow.executors.celery_executor.app.)
- No workers listening on the queue a task uses (
queue='etl_heavy') → tasks queue forever. Start workers with that queue:airflow celery worker -q etl_heavy. - Broker (Redis/RabbitMQ) restarted and lost messages → tasks stay
queuedin the DB but no message exists. Step 5.
3. Kubernetes executor: can pods start?
kubectl get pods -n airflow --field-selector=status.phase=Pending
kubectl describe pod <pod> -n airflow | sed -n '/Events/,$p'
kubectl describe resourcequota -n airflow
Fix quota, node capacity or image pull problems; the scheduler then creates pods for queued tasks.
4. Limits that block scheduling
airflow config get-value core parallelism
airflow config get-value core max_active_tasks_per_dag
airflow config get-value core max_active_runs_per_dag
airflow pools list
parallelismcaps running tasks across the whole installation (per scheduler).- A backfill with many active runs can fill
max_active_runs_per_dagand pool slots, starving production DAGs. - Pools: a pool with 8 slots and 8 long-running tasks blocks everything else in it.
Quick relief:
airflow pools set etl_pool 32 "INC12345 temporary increase"
airflow dags pause backfill_events_2025 # pause the noisy backfill
Warning
Raising pool slots and parallelism lets more work hit downstream databases at once. Increase in steps and watch the sources.
5. Clear zombie queued tasks
Tasks stuck in queued with no executor record can be reset so the scheduler queues them again:
airflow tasks clear my_dag -t extract_orders -s 2026-10-31 -e 2026-10-31 --only-running --yes
Airflow 2.x also requeues tasks stuck in queued after scheduler.task_queued_timeout (newer versions; check yours). Lowering it temporarily speeds up recovery.
Restarting the scheduler(s) re-adopts or resets orphaned tasks:
kubectl rollout restart deployment/airflow-scheduler -n airflow
6. Verify
Watch running task count rise and queued count fall. Prioritise critical DAGs with priority_weight if capacity is limited.
Prevent it next time
- Separate pools for production and backfills; cap backfill
max_active_runs. - Alerts on queued task age and on number of tasks in
queued. - Make worker queues explicit in deployment config so no queue is left without workers.
- Keep the broker persistent (Redis AOF / RabbitMQ durable queues) where possible.