Standalone Apache Airflow on one VM: a supportable Docker Compose setup

The official docker-compose.yaml is designed for learning (CeleryExecutor, Redis, a bundled Postgres). For a small production workload, simplify and harden it.

#1. Architecture

VM (8 vCPU, 32 GB)
├── airflow-scheduler (LocalExecutor, runs tasks as subprocesses)
├── airflow-webserver / api-server
├── airflow-dag-processor (Airflow 3 runs it separately)
└── triggerer (if you use deferrable operators)
External: PostgreSQL (managed: RDS / Azure Flexible Server / OCI PostgreSQL), object storage for logs

LocalExecutor is enough for tens of concurrent tasks and removes Redis and Celery workers.

#2. Key settings (environment)

x-airflow-env: &airflow-env
  AIRFLOW__CORE__EXECUTOR: LocalExecutor
  AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:${AF_DB_PASS}@pg-airflow.internal:5432/airflow
  AIRFLOW__CORE__PARALLELISM: "32"
  AIRFLOW__CORE__LOAD_EXAMPLES: "false"
  AIRFLOW__LOGGING__REMOTE_LOGGING: "true"
  AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "s3://airflow-logs-prod/logs"
  AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "aws_logs"
  AIRFLOW__SECRETS__BACKEND: "airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend"

• Use an external metadata database with backups and PITR, not a Postgres container on the same VM.
• Remote logging to object storage so logs survive VM rebuilds.
• A secrets backend (AWS Secrets Manager, Azure Key Vault, HashiCorp Vault) instead of connections stored in the metadata DB.

#3. DAG deployment

DAGs from Git: a CI job that syncs the repo to the VM (or a git-sync sidecar), never editing files on the server.

#4. Operations

docker compose up -d
docker compose ps
docker compose exec airflow-scheduler airflow jobs check --job-type SchedulerJob
docker compose exec airflow-scheduler airflow db check

• airflow db clean --clean-before-timestamp '2026-07-01' --yes monthly to keep the metadata DB small (back up first).
• Upgrade by changing the image tag in a test copy, running airflow db migrate, then production.

#5. Monitoring

• Health endpoint (/health in Airflow 2, /api/v2/monitor/health in Airflow 3) for scheduler heartbeat.
• StatsD/OpenTelemetry metrics to your monitoring stack.
• Alert on: scheduler heartbeat missing, DAG import errors, task failures for critical DAGs.

#6. Backup and recovery

• Metadata DB: managed backups + PITR.
• DAGs: in Git.
• Config: compose file and .env in Git (secrets excluded).
• Recovery = new VM + docker compose up pointing to the same database. Test it once.

#Limits

One VM is a single point of failure; tasks stop while it's down (the scheduler catches up after restart according to catchup settings). When downtime isn't acceptable, move to multiple schedulers on Kubernetes or a managed service (MWAA, Cloud Composer, Astronomer). See the Airflow HA post.