Sev1: Airflow scheduler down or stuck — metadata database full, locked or out of connections

#Symptoms

• UI banner: "The scheduler does not appear to be running".
• DAG runs stay queued/scheduled; nothing executes.
• Scheduler logs show OperationalError, too many connections, lock timeouts or very slow loops.

#1. Is the scheduler alive?

airflow jobs check --job-type SchedulerJob --limit 1
kubectl logs deploy/airflow-scheduler -n airflow --tail=200      # Helm chart deployment

The health endpoint (/health on the webserver) reports the scheduler's latest heartbeat.

#2. Is the metadata database healthy?

PostgreSQL:

SELECT count(*), state FROM pg_stat_activity WHERE datname = 'airflow' GROUP BY state;
SHOW max_connections;
SELECT pg_size_pretty(pg_database_size('airflow'));
SELECT relname, n_live_tup, n_dead_tup FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 10;
SELECT pid, wait_event_type, wait_event, now() - query_start AS running, left(query, 80)
FROM pg_stat_activity WHERE datname = 'airflow' AND state <> 'idle' ORDER BY running DESC LIMIT 10;

#3. Clear blockers

SELECT pg_terminate_backend(pid) FROM pg_stat_activity
WHERE datname = 'airflow' AND state = 'idle in transaction' AND now() - state_change > interval '10 minutes';

Restart the scheduler(s) after the DB is healthy:

kubectl rollout restart deploy/airflow-scheduler -n airflow

#4. Clean old metadata

airflow db clean --clean-before-timestamp "2026-07-25" --tables log,xcom,job,dag_run,task_instance --dry-run
airflow db clean --clean-before-timestamp "2026-07-25" --tables log,xcom,job,dag_run,task_instance --yes

By default it archives rows into _airflow_deleted__* tables; drop those after checking (airflow db drop-archived). Run during a quiet period — deleting millions of rows is heavy. Then VACUUM ANALYZE.

#5. Get the backlog through safely

When scheduling resumes, every queued run starts at once. Protect downstream systems:

• Temporarily lower max_active_runs on heavy DAGs, or pause less critical DAGs.
• Use pools to limit concurrent tasks against shared resources (databases, APIs).
• For DAGs with catchup=True, decide whether missed intervals should run or be marked success.

airflow dags pause reporting_hourly
airflow dags unpause reporting_hourly      # later

#Prevent it next time

• Scheduled airflow db clean (weekly) with a sensible retention.
• PgBouncer in front of the metadata DB; alerts on connections, disk and scheduler heartbeat age.
• Two schedulers (HA, Airflow 2+) so one stuck process isn't an outage.
• Avoid heavy top-level code in DAG files — slow parsing starves the scheduler loop.