Applies toApache Airflow (self-managed, MWAA, Cloud Composer, Astronomer), Azure Data Factory / Fabric Data Factory pipelines, Databricks Workflows (Lakeflow Jobs), Dagster, Prefect, dbt Cloud jobs
The options at a glance
| Airflow | ADF / Fabric pipelines | Databricks Workflows | Dagster / Prefect | |
|---|---|---|---|---|
| Style | Python code (DAGs) | Visual + JSON, low-code | UI/YAML/API, platform-native | Python code (assets/flows) |
| Best at | Orchestrating many systems | Data movement and Azure integration | Jobs inside Databricks | Asset-centric pipelines, developer experience |
| Connectors | Huge provider ecosystem | Many built-in connectors, SHIR for on-prem | Databricks tasks, dbt, SQL, notebooks | Integrations library |
| Hosting | Self-managed or managed services | SaaS | SaaS (in workspace) | OSS or SaaS |
| Who builds | Engineers | Engineers + analysts | Databricks users | Engineers |
Decision questions
- What do you orchestrate? Mostly Databricks jobs → Workflows. Mostly copying data into Azure → ADF/Fabric. Many different systems (APIs, warehouses, SaaS, scripts) → Airflow or Dagster/Prefect.
- Who builds pipelines? Python engineers → code-first. Mixed teams → visual tools or a platform-native option.
- Operations capacity? Self-managed Airflow is a platform to run. Managed or SaaS options reduce that.
- Lineage and assets? Asset-centric orchestration (Dagster, Airflow Assets, Databricks lineage) helps reason about data, not just tasks.
- CI/CD: Can pipelines be versioned, tested and promoted easily? (See the ADF Git/CI transformation post.)
Common patterns
| Pattern | Notes |
|---|---|
| ADF for ingestion → Databricks Workflows for transformation | Fine. Let ADF trigger the Databricks job and wait, or use events |
| Airflow as the "orchestrator of orchestrators" | Airflow triggers ADF pipelines, Databricks jobs, dbt Cloud jobs and waits with deferrable operators |
| Platform-native only | Simple if everything lives on one platform |
| Mixed with no owner | Pipelines triggered from several places; nobody knows the end-to-end flow (avoid) |
Avoid these traps
- Two schedulers triggering the same job ("belt and braces"). Duplicate runs (see the duplicates Sev1 post).
- Orchestrator running heavy compute itself (pandas in Airflow workers). Push work to the data platform.
- Hard-coded credentials in pipeline definitions. Use managed identities/secret backends.
- No dependency on data readiness, only time-based schedules. Prefer events, sensors or asset triggers.
Operating needs to check
| Need | Ask |
|---|---|
| Alerting | Failures, SLA misses, long-running tasks to on-call channels |
| Backfills | How easy and safe is re-running a date range? |
| Observability | Run history, logs retention, lineage |
| Access control | Who can trigger, edit, view secrets? |
| HA | What happens if the scheduler or region fails? |
Tip
Pick one primary orchestrator for cross-system flows and let platform-native schedulers handle work inside their platform. Document which system owns which trigger.
Checklist
- What is orchestrated and who builds it is clear
- Hosting and operations model chosen
- CI/CD and testing approach confirmed
- Ownership of each trigger documented
- Alerting, backfill, observability and HA verified