Home / Data Platform / Airflow

Data PlatformHow-toAdvanced

Airflow high availability: multiple schedulers, an HA metadata DB and resilient workers

Run Airflow without single points of failure using multiple active schedulers, a highly available PostgreSQL metadata database, redundant webservers and workers on Kubernetes, and remote logs.

Deepak Bhardwaj2 min read

Applies toApache Airflow 2.x/3.x on Kubernetes (official Helm chart); concepts apply to VMs

Components and their HA approach

Component HA approach
Scheduler 2+ active schedulers (supported since Airflow 2.0; they coordinate through row locks in the DB)
Metadata database Managed PostgreSQL with HA (Multi-AZ / zone-redundant / Patroni)
Webserver / API server 2+ replicas behind a load balancer
Workers Celery workers (2+, across zones) or KubernetesExecutor pods
DAG processor (Airflow 3) / triggerer 2+ replicas
Broker (Celery) Managed Redis with replication, or use KubernetesExecutor to avoid a broker
Logs Remote logging to object storage

Helm values (excerpt)

executor: CeleryKubernetesExecutor
scheduler:
  replicas: 2
  podDisruptionBudget: { enabled: true, config: { maxUnavailable: 1 } }
webserver:
  replicas: 2
workers:
  replicas: 3
triggerer:
  replicas: 2
postgresql:
  enabled: false          # use an external HA database
data:
  metadataConnection:
    user: airflow
    host: pg-airflow-prod.internal
    db: airflow
    sslmode: require
pgbouncer:
  enabled: true

Spread pods across zones (topology spread or anti-affinity) and use PDBs.

Metadata database requirements

  • PostgreSQL with automatic failover (RDS Multi-AZ, Azure Flexible Server zone-redundant HA, Cloud SQL HA).
  • PgBouncer in front: multiple schedulers and many workers open many connections.
  • Supports SELECT ... FOR UPDATE SKIP LOCKED (PostgreSQL 12+ and MySQL 8+ do; that's how schedulers coordinate).

Task-level resilience

  • Retries with backoff and idempotent tasks (worker loss = task retried).
  • task_instance_heartbeat_timeout / zombie detection tuned so lost workers are noticed quickly.
  • Celery: task_acks_late so tasks from a dead worker are redelivered.

What happens on failures

Failure Effect
One scheduler pod dies The other keeps scheduling; Kubernetes restarts the pod
DB failover Short pause (seconds to a minute); schedulers reconnect
Worker node lost Running tasks fail/zombie and get retried elsewhere
Zone outage Pods reschedule to other zones if capacity exists

Monitoring

  • Scheduler heartbeat and scheduler.tasks.starving, executor.open_slots, dag_processing.import_errors metrics.
  • Alert if no scheduler heartbeat for 2 minutes.

Tip

More schedulers don't fix a slow database. If scheduling lags, check DB CPU and connection counts first, and tune max_dagruns_to_create_per_loop and DAG parsing before adding a third scheduler.

Related fixes