Airflow tasks stuck in queued or scheduled - finding the bottleneck in pools, concurrency limits, executors and workers

#Know the states

scheduled and queued point to different bottlenecks. Count them first:

airflow tasks states-for-dag-run <dag_id> <run_id>

Or across all DAGs, in the metadata database:

SELECT state, count(*) FROM task_instance
WHERE state IN ('scheduled','queued','running') GROUP BY state;

#Stuck in scheduled: limits

Airflow checks several limits before it queues a task. Any one can block.

Check pools:

airflow pools list

A pool showing 0 open slots with many queued tasks is the bottleneck. The default_pool is often left at its default size, which is too small for real use.

airflow pools set default_pool 128 "Default pool"

Check the effective config values:

airflow config get-value core parallelism
airflow config get-value core max_active_tasks_per_dag

Long-running sensors in poke mode hold a slot the whole time they wait. Switch them to mode="reschedule", or use deferrable operators with a triggerer, to free slots.

#Stuck in scheduled: the scheduler

airflow jobs check --job-type SchedulerJob --allow-multiple --limit 5

If no healthy scheduler job is found, the scheduler isn't heartbeating. Check its logs for database errors or DAG parsing timeouts. Very slow DAG parsing (heavy top-level code) delays scheduling too:

airflow dags report      # parse time per file

Move API calls, database queries and large imports out of DAG top-level code.

#Stuck in queued: executor and workers

CeleryExecutor

celery --app airflow.providers.celery.executors.celery_executor.app inspect active
celery --app airflow.providers.celery.executors.celery_executor.app inspect stats

(On older Airflow versions the app path is airflow.executors.celery_executor.app.)

• No workers respond: workers are down or can't reach the broker (Redis/RabbitMQ).
• Workers are busy: each worker runs celery.worker_concurrency tasks. Add workers or raise concurrency within the memory limits.
• Tasks routed to a queue= that no worker listens on stay queued for ever. Check the operator's queue argument against the workers' -q option.

KubernetesExecutor

kubectl get pods -n airflow --field-selector status.phase=Pending
kubectl describe pod <task-pod> -n airflow | sed -n '/Events/,$p'

Pending pods point to cluster capacity, resource requests that are too large, image pull errors or quota limits. Those are Kubernetes problems rather than Airflow problems.

Managed services

On MWAA or Composer, check the worker count and autoscaling limits in the environment settings, and the service's own metrics for queued tasks.

#Tasks that were queued and then got lost

If a worker died, tasks can remain queued or running with no process behind them. The scheduler eventually detects zombies and orphaned tasks. To clear specific ones:

airflow tasks clear <dag_id> -t <task_regex> -s 2026-10-26 -e 2026-10-26 --only-running --yes

Warning

Clearing reruns the tasks. Check that they're idempotent, or that the downstream system can handle a second write, before clearing production runs.

#Quick decision path

1. Most tasks scheduled? → check pools, then DAG limits, then parallelism, then scheduler health.
2. Most tasks queued? → check workers, broker or pods.
3. One DAG only? → its max_active_tasks, max_active_runs or its pool.
4. One task only? → its queue, pool or max_active_tis_per_dag.

#Prevent it next time

• Size pools per downstream system (for example a warehouse pool sized to its connection limit) rather than relying on default_pool.
• Use deferrable operators or reschedule-mode sensors.
• Alert on the scheduler.tasks.starving and executor.queued_tasks metrics (names vary by version; check your StatsD/OpenTelemetry output) and on the age of the oldest queued task.
• Load test before adding a large backfill.

#Checklist

• Counted tasks by state
• Pools and DAG/task limits checked
• Scheduler heartbeat and DAG parse time checked
• Workers/pods healthy and listening on the right queues
• Sensors in reschedule or deferrable mode
• Alerts on queued and starving tasks