Symptoms
- Tasks sit in
scheduled/queuedfor minutes while workers are idle. - DAG changes take a long time to appear.
- Sensors occupy most worker slots.
- The UI is slow and the metadata DB CPU is high.
1. Cheap DAG parsing
DAG files are parsed repeatedly. Top-level code runs on every parse.
| Avoid at top level | Instead |
|---|---|
| Database/API calls to build task lists | Read a config file shipped with the DAG, or Variables/Assets used inside tasks |
| Heavy imports (pandas, ML libs) | Import inside the task callable |
Variable.get() in loops | Jinja templating ({{ var.value.x }}) or a single read |
Check parse times:
airflow dags report # Airflow 2.x: parse duration per file
Airflow 3 runs the DAG processor as a separate component. Check airflow dags report / processor logs and the [dag_processor] settings for your version. Tune parse intervals so changes appear fast enough without constant re-parsing (e.g. min_file_process_interval, refresh_interval; names differ by version).
2. Concurrency settings, top to bottom
| Level | Setting (names vary slightly by version) | Purpose |
|---|---|---|
| Installation | core.parallelism | Max running task instances overall |
| DAG | max_active_tasks | Max running tasks per DAG |
| DAG | max_active_runs | Concurrent DAG runs (backfills!) |
| Task | max_active_tis_per_dag | Limit a specific task across runs |
| Executor | Celery worker_concurrency, K8s pod limits | Slots per worker |
| Pool | Pool slots | Protect shared resources |
Find the limiting one: if parallelism is 32 but you have 100 worker slots, tasks queue for no reason. If max_active_runs is high, a backfill can flood everything.
3. Pools protect downstream systems
airflow pools set warehouse_heavy 4 "Max 4 concurrent heavy warehouse loads"
airflow pools set erp_api 2 "ERP API allows 2 concurrent sessions"
load = SQLExecuteQueryOperator(task_id="load_orders", sql="...", pool="warehouse_heavy", priority_weight=10)
Use priority_weight so critical DAGs get slots first when pools are full.
4. Deferrable operators instead of blocking sensors
A classic sensor in poke mode holds a worker slot while waiting. Deferrable operators hand the wait to the triggerer and free the slot:
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
wait_file = S3KeySensor(task_id="wait_for_file", bucket_name="landing", bucket_key="orders/{{ ds }}/_SUCCESS",
deferrable=True, timeout=6 * 3600)
Run at least one triggerer (two for HA). Where no deferrable version exists, use mode="reschedule" with a sensible poke_interval.
5. Keep the metadata database lean
airflow db clean --clean-before-timestamp "$(date -d '-60 days' +%F)" --skip-archive --yes
Run it on a schedule (it's a maintenance DAG in many setups). Large task_instance, log, xcom and dag_run tables slow the scheduler and UI. Don't pass large data through XCom; use object storage and pass references.
6. Scheduler and executor capacity
- Run two or more schedulers for HA and throughput (Airflow 2+ supports active-active schedulers with a supported database).
- KubernetesExecutor: pod start time adds latency per task. For many tiny tasks, CeleryExecutor or the hybrid executor setup may suit better.
- Database connection pooling (PgBouncer) for many schedulers/workers.
7. Design for fewer, bigger tasks
Thousands of tiny tasks per run stress the scheduler. Batch small operations into one task where failure isolation isn't needed, and use dynamic task mapping with sensible chunk sizes.
Tip
Measure scheduling delay (task
start_dateminus when it became runnable) as a metric. It tells you whether tuning helps far better than looking at the UI.
Checklist
- No heavy top-level code; parse times checked
- Concurrency limits reviewed top to bottom
- Pools for shared systems with priorities
- Deferrable operators / reschedule mode for waits
- Metadata DB cleaned on a schedule; no big XComs
- Scheduler HA and executor fit reviewed