Applies todbt tests, Great Expectations / Soda-style checks, Databricks Lakeflow expectations, Snowflake data metric functions, data observability tools
The layers of checks
| Layer | What to check | Example |
|---|---|---|
| Ingestion (bronze) | Arrival, volume, schema | File arrived by 06:00; row count within normal range; no new/missing columns |
| Cleansing (silver) | Validity, uniqueness, referential integrity | order_id unique and not null; customer_id exists in customers |
| Business (gold) | Business rules, reconciliation | Daily revenue matches the finance system within tolerance |
| Consumption | Freshness and availability | Dashboard dataset refreshed within SLA |
Rules vs monitoring
| Rule-based tests | Anomaly monitoring |
|---|---|
| Known expectations (not null, unique, accepted values) | Unknown problems (volume drops, distribution shifts) |
| Written by engineers with the business | Learned from history |
| Precise, few false positives if well written | Catches surprises; needs tuning to avoid noise |
Use both: rules for what you know must be true, monitoring for what you didn't think of.
# dbt example: severity decides block vs warn
models:
- name: fct_orders
columns:
- name: order_id
data_tests:
- unique
- not_null
- name: amount
data_tests:
- dbt_utils.accepted_range:
min_value: 0
config:
severity: warn
Block or warn?
| Failure | Action |
|---|---|
| Primary key duplicates, nulls in keys, broken joins | Block publishing to gold; keep last good data |
| Out-of-range values in a few rows | Quarantine bad rows, publish the rest, alert |
| Volume or distribution anomaly | Warn and investigate; block only for critical tables |
| Late data | Alert per freshness SLA |
Blocking protects consumers but delays data. Agree the rule per table with its owner.
Ownership and routing
- Every critical dataset has an owner team and an alert channel.
- Alerts carry context: table, check, failing sample, recent runs, runbook link.
- Source-caused failures go to the source owner (see the ingestion source contract idea).
- Track incidents to closure. A failing test that stays red for weeks should be fixed or removed.
Measure quality simply
| Indicator | Definition |
|---|---|
| Freshness SLA met | % of days critical tables met their deadline |
| Test pass rate | % of critical checks passing |
| Incidents | Count and time to resolve data incidents |
| Consumer-reported issues | Problems found by users before the team (aim to reduce) |
Tip
Start with freshness, volume and key uniqueness on the 20 most-used tables. That catches most real incidents before you invest in hundreds of detailed rules.
Checklist
- Checks placed per layer
- Rules and anomaly monitoring both in use
- Block/warn/quarantine decided per critical table
- Owners and alert routing defined
- Quality indicators reported
- Stale failing tests reviewed regularly