Sev1: Lakeflow Declarative Pipeline (DLT) failed in production — recover without a full refresh

#Symptoms

• Pipeline updates fail; gold tables and dashboards stop updating.
• Error messages mention schema changes, expectation failures, checkpoint problems or cluster launch failures.
• Someone suggests "just do a full refresh".

#1. Read the event log

In the pipeline UI open the failed update and the error. For detail, query the event log:

SELECT timestamp, level, message, details
FROM event_log(TABLE(prod.sales.orders_silver))
WHERE level IN ('ERROR','WARN')
ORDER BY timestamp DESC
LIMIT 50;

(You can also use event_log('<pipeline-id>').)

#2. Why not a full refresh

Caution

A full refresh truncates streaming tables and reprocesses from the source. If the source no longer holds all history (Event Hubs/Kafka retention, files moved or deleted), data is lost permanently. Before any full refresh, confirm the source can replay everything.

Protect key tables from accidental full refresh:

@dlt.table(table_properties={"pipelines.reset.allowed": "false"})
def orders_bronze():
    ...

#3. Fix by error type

Schema change: for Auto Loader sources, allow schema evolution or add rescued data:

spark.readStream.format("cloudFiles") \
  .option("cloudFiles.format", "json") \
  .option("cloudFiles.schemaEvolutionMode", "addNewColumns") \
  .option("rescuedDataColumn", "_rescued_data") \
  .load(path)

With addNewColumns the stream stops once when it sees new columns and continues on the next update.

Expectation failures: switch the strict rule to drop or quarantine during the incident, then fix the data upstream:

@dlt.expect_or_drop("valid_amount", "amount >= 0")

Run-as / permissions: check the pipeline's run-as principal still has USE CATALOG, USE SCHEMA, SELECT on sources and CREATE TABLE/MODIFY on targets.

#4. Refresh only what is needed

Use selective refresh. Through the UI ("Select tables for refresh") or the API:

databricks pipelines start-update <pipeline-id> --json '{
  "refresh_selection": ["prod.sales.orders_silver","prod.sales.orders_gold"]
}'

Full refresh of a single non-streaming (materialized view) table is safe because it recomputes from upstream tables:

databricks pipelines start-update <pipeline-id> --json '{"full_refresh_selection": ["prod.sales.daily_revenue_mv"]}'

#5. Incompatible streaming state

If a code change (for example a changed aggregation or join) breaks the streaming state of one table, full refresh that table only, and only after confirming its source can replay. For a bronze table reading from Kafka/Event Hubs with short retention, instead create a new table that starts from now and backfill history from the old table.

#6. Verify

• Update completes; check row counts and freshness on gold tables.
• Check data quality metrics in the event log for dropped rows during the incident and report them.

#Prevent it next time

• Set pipelines.reset.allowed = false on bronze/streaming tables fed by non-replayable sources.
• Separate dev and prod pipelines; promote with Databricks Asset Bundles.
• Use addNewColumns and rescued data columns at bronze.
• Alert on update failures and table freshness.