Applies toLakeflow Declarative Pipelines (Delta Live Tables), Lakeflow Jobs (Workflows) with notebooks/Python wheels, dbt Core/Cloud with the dbt-databricks adapter
The three approaches
| Declarative pipelines | Jobs with notebooks/files | dbt on Databricks | |
|---|---|---|---|
| Style | Declare tables; engine handles dependencies, incremental processing, retries | Write the steps yourself in PySpark/SQL | SQL models with Jinja; dependency graph from ref() |
| Strengths | Streaming + batch in one model, CDC (AUTO CDC), expectations, managed maintenance | Full flexibility, any library, complex logic | Analytics-engineer friendly, testing, docs, portable SQL |
| Weak spots | Less control over execution details | You build incremental logic, retries, quality checks | Python logic limited; streaming limited |
| Typical team | Data engineers | Data engineers, ML engineers | Analytics engineers, SQL-first teams |
Map use cases
| Use case | Default |
|---|---|
| Ingestion from files/Event Hubs into bronze/silver | Declarative pipelines (or Auto Loader jobs) |
| CDC into SCD1/SCD2 tables | Declarative pipelines AUTO CDC |
| Business marts (gold) by SQL-first teams | dbt or declarative pipelines in SQL |
| Complex Python (APIs, ML features, custom parsing) | Jobs with Python files/wheels |
| Orchestrating mixed steps | Lakeflow Jobs calling pipelines, dbt tasks and Python tasks |
Decision questions
- What skills does the team have (SQL, PySpark, dbt)?
- Is streaming or CDC needed?
- How much control over execution and libraries do you need?
- How will you test, deploy and monitor? (All three work with Asset Bundles/CI.)
- Do you need portability of transformation logic to other platforms? (dbt is the most portable.)
Mixing them well
- One orchestrator (Lakeflow Jobs) with tasks: pipeline task → dbt task → Python task.
- Clear layer ownership: e.g. pipelines own bronze/silver, dbt owns gold.
- Shared conventions for catalogs, schemas, naming and tags.
Avoid
- The same table written by both a pipeline and a dbt model.
- Copy-pasted notebook logic across jobs (package as wheels; see the wheel post).
- A different approach per developer.
Tip
Pick a default per layer and write it down. "Bronze/silver in declarative pipelines, gold in dbt, custom Python only with justification" is clear enough to stop drift.
Checklist
- Team skills and needs assessed
- Default approach per layer chosen
- Orchestration via one job layer
- Table ownership per approach clear
- CI/CD covers all approaches