Applies toPipelines on Databricks, Snowflake, Fabric, ADF, Airflow, dbt; monitoring with Azure Monitor, CloudWatch, Datadog, Grafana or data observability tools
SLIs that matter for data
| SLI | Measure |
|---|---|
| Freshness | Time between now and the latest loaded data (or when the dataset was last published) |
| Timeliness | Dataset published by the agreed time (e.g. 06:00) |
| Completeness | Rows/partitions loaded vs expected (or vs source counts) |
| Correctness | Critical data quality checks passing |
| Availability | Query endpoint/warehouse/semantic model reachable |
Write SLOs per critical dataset
| Dataset | SLO | Consumers | Owner |
|---|---|---|---|
gold.daily_sales | Published by 06:00 on 99% of business days | Sales leadership dashboard | Sales data team |
silver.orders | Freshness ≤ 15 min, 99.5% of the time during business hours | Ops dashboard, fraud review | Checkout data team |
| Finance month-end mart | Complete and reconciled by day 2, 06:00 | Finance close | Finance data team |
Not every table needs an SLO. Pick the ones people would escalate about.
Three kinds of signals
| Kind | Examples | Source |
|---|---|---|
| Pipeline | Run status, duration, retries, queue time | Orchestrator, job APIs, system tables |
| Platform | Cluster/warehouse saturation, capacity throttling, broker lag | Platform metrics |
| Data | Freshness timestamps, row counts, test results | Queries on the data, quality tools |
Bring them together so an alert says "daily_sales at risk: upstream job 40 min slower than usual, warehouse queueing", not three separate alerts.
-- Simple freshness SLI query (adapt per platform)
SELECT 'gold.daily_sales' AS dataset,
MAX(_loaded_at) AS last_load,
DATEDIFF(minute, MAX(_loaded_at), CURRENT_TIMESTAMP) AS minutes_stale
FROM gold.daily_sales;
Alert on SLO risk
| Instead of | Alert on |
|---|---|
| Every task failure (retries often fix them) | Dataset will miss its deadline (projected or actual) |
| Long-running job at a fixed threshold | Duration far above its usual range |
| Each failing test | Critical checks failing on SLO datasets |
Lower-priority signals go to a dashboard or daily digest, not the pager.
Review with consumers
- Monthly: SLO attainment per dataset, misses with causes, actions.
- Use misses to drive engineering work (flaky sources, capacity, slow models).
- Adjust SLOs when the business need changes, both up and down.
Tip
Publish a simple status page or dashboard showing each critical dataset's freshness and SLO state. It cuts "is the data ready?" messages dramatically.
Checklist
- Critical datasets and consumers identified
- SLOs written with owners
- Pipeline, platform and data signals collected together
- Alerts based on SLO risk, not every failure
- Status visible to consumers
- Monthly SLO review in place