Home / Data Platform / dbt & Quality
dbt projects, incremental models, CI and data quality testing. 10 posts.
Set up a dbt Core project, declare sources, build staging and mart models with ref(), add built-in tests, choose materializations, and generate documentation, with the commands you will run every day.
dbt snapshots record how source rows change over time as SCD type 2 history. Pick the timestamp or check strategy, set a stable unique key, handle hard deletes, and avoid the mistakes that corrupt history.
Data quality work fails when it is a pile of tests nobody acts on. Place checks at the right layer, mix rule-based tests with anomaly monitoring, decide which failures block a pipeline and which only warn, route alerts to owners, and measure quality with a few clear indicators.
A dbt project that grows without structure becomes slow to build and hard to change. Use clear staging, intermediate and mart layers with naming rules, decide between a monolith and several projects with cross-project references, set ownership and access per group, and keep CI and environments consistent.
dbt projects get slower and more expensive as they grow because everything is rebuilt as tables on every run and CI rebuilds the world. Find the slowest models, make big ones incremental with the right strategy, pick materialisations deliberately, use slim CI with state comparison and tune threads and warehouse sizes.
Recover an incremental dbt model that lost years of history because someone ran --full-refresh against a source that only keeps recent data. Restore from warehouse time travel, protect the model with full_refresh=false and fix the run process.
Replace mapping-heavy ETL tools with SQL-first ELT in dbt. Inventory mappings, separate extraction from transformation, rebuild logic as layered dbt models with tests, run old and new in parallel and retire the ETL servers.
The dbt selectors, graph operators and CLI flags you need on support duty to rebuild one model, its downstream, only what changed, or only what failed.
A practical set of freshness, volume, schema, uniqueness and business-rule checks for pipelines, with SQL and dbt examples and guidance on blocking vs warning.
Separate compilation, connection, database and test failures in dbt, use the compiled SQL and logs to debug, and rerun only what failed.
No posts in this category yet.