Why move
Self-run Airflow means patching, scaling workers, nursing the metadata DB and upgrading Airflow yourself. Managed services handle the platform; you keep the DAGs.
1. Prepare the DAGs
- Version: pick the target Airflow version supported by the service and test DAGs on it locally (
astro dev startor the MWAA local runner image). - Dependencies: pin everything in
requirements.txt(with the Airflow constraints file). Managed services restrict OS packages; replace those dependencies or run them in containers (KubernetesPodOperator/ECS operator). - Deprecated imports: fix
airflow.contrib, old provider paths and deprecated operators. - Top-level code: remove DB calls and heavy imports at DAG parse time.
- Local files: anything reading local disk on the old server must use object storage.
airflow dags list-import-errors # on the target test environment
2. Connections and variables
Move them out of the metadata DB into a secrets backend:
airflow connections export connections.json
airflow variables export variables.json
Load them into AWS Secrets Manager / Key Vault with the naming convention the backend expects (for example airflow/connections/<conn_id>), then configure:
secrets.backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
secrets.backend_kwargs = {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}
Never commit the exported files.
3. Build the environment
- Networking: private web server access, workers that can reach your databases and APIs (VPC/VNet, NAT, private endpoints).
- Sizing: environment class/worker counts from your current concurrency (
parallelism, pools, peak running tasks). - Configuration overrides:
core.parallelism,celery.worker_concurrency, pools — recreate pools explicitly.
4. History
Most teams don't migrate metadata history. Start fresh and archive old logs. If you need history for audit, keep the old DB read-only or export run data to a table.
5. Switch DAG by DAG
- Deploy all DAGs to the new environment paused.
- For each DAG: pause on the old server after its last run, unpause on the new one. Set
start_date/catchupso it doesn't backfill history unexpectedly. - Watch the first runs; compare task durations and outputs.
Warning
Never run the same DAG unpaused in both environments — you'll double-load data. Use a checklist per DAG with who switched it and when.
6. CI/CD
DAG deployment from Git: sync to the MWAA S3 bucket, astro deploy, or Composer's bucket, after running dags test/import checks in CI.
Checklist
- DAGs import cleanly on the target version
- Requirements pinned with constraints; no OS-level dependencies
- Connections/variables in a secrets backend
- Network access from workers verified
- DAGs switched one by one; old server retired