Sev1: a dev or test job wrote into production tables. Contain, find the damage and restore with Delta history

#Symptoms

• Production dashboards show test values ("test_customer", dates in 1970, duplicated totals).
• DESCRIBE HISTORY on a production table shows a write from a user or job that should never write there.
• A deploy changed a bundle variable or a widget default (for example catalog = sales_prod) in a non-prod target.

#Step 1: stop the writer (5 minutes)

Find the job or notebook from the table history:

DESCRIBE HISTORY sales_prod.silver.orders LIMIT 20;

The userName, job (job ID, run ID) and notebook columns identify the writer. Pause it:

SCHED=$(databricks jobs get <job-id> -o json | jq -c '.settings.schedule | .pause_status = "PAUSED"')
databricks jobs update <job-id> --json "{\"new_settings\": {\"schedule\": $SCHED}}"
databricks jobs cancel-all-runs <job-id>

Also suspend downstream production jobs that would propagate the bad data to gold and BI.

Tip

Quick containment for a person: revoke MODIFY on the production schema from the group or principal that made the write, then put it back after the fix.

REVOKE MODIFY ON SCHEMA sales_prod.silver FROM `sales-engineers`;

#Step 2: find every table it touched

Delta history only covers one table at a time. Use lineage and audit to find all of them:

SELECT DISTINCT target_table_full_name, entity_type, entity_id, event_time
FROM system.access.table_lineage
WHERE event_time > current_timestamp() - INTERVAL 2 DAYS
  AND target_table_catalog = 'sales_prod'
  AND (entity_id = '<job-id>' OR created_by = 'alice@contoso.com')
ORDER BY event_time;

Column names in system tables evolve; check DESCRIBE system.access.table_lineage if a column is missing. Cross-check with the job's code: every saveAsTable, MERGE INTO and INSERT target.

For each table, record the bad version(s):

SELECT version, timestamp, operation, operationParameters, operationMetrics, userName, job
FROM (DESCRIBE HISTORY sales_prod.silver.orders)
WHERE timestamp > '2026-10-29T00:00:00'
ORDER BY version;

#Step 3: decide restore strategy per table

Restore:

RESTORE TABLE sales_prod.silver.orders TO VERSION AS OF 4810;

Remove only bad rows by comparing with a good version:

CREATE OR REPLACE TEMP VIEW bad_rows AS
SELECT * FROM sales_prod.silver.orders VERSION AS OF 4812
EXCEPT ALL
SELECT * FROM sales_prod.silver.orders VERSION AS OF 4811;

-- inspect first
SELECT count(*), min(order_date), max(order_date) FROM bad_rows;

MERGE INTO sales_prod.silver.orders t
USING (SELECT DISTINCT order_id FROM bad_rows) b
ON t.order_id = b.order_id
WHEN MATCHED THEN DELETE;

That works when the bad write added new keys. If it updated existing keys, restore the old values from the good version with MERGE instead of deleting.

Caution

Time travel depends on files not yet removed by VACUUM (default 7 days). If predictive optimization or a scheduled VACUUM runs before you finish, the old versions may disappear. Take a deep clone of the last good version now: CREATE TABLE sales_prod.recovery.orders_v4811 DEEP CLONE sales_prod.silver.orders VERSION AS OF 4811;

#Step 4: repair downstream

• Rerun gold jobs for the affected dates (with replaceWhere or MERGE).
• Refresh materialized views and dashboards.
• Tell report owners which period was affected and when it was corrected.

#Step 5: confirm

• Row counts and key metrics match expected values (compare with the restored version and source systems).
• DESCRIBE HISTORY shows the restore/repair operations with your incident ID in a comment if you used SET spark.databricks.delta.commitInfo.userMetadata = 'INC-1234' before writing.
• Writers re-enabled only after step 6 is in place.

#Step 6: make it impossible next time

• Workspace-catalog binding: bind sales_prod to the production workspace only. Dev workspaces then cannot see it at all.

databricks catalogs update sales_prod --isolation-mode ISOLATED
databricks workspace-bindings update-bindings catalog sales_prod --json '{"add": [{"workspace_id": 1234567890123456, "binding_type": "BINDING_TYPE_READ_WRITE"}]}'

• Humans get read-only on production catalogs; only production service principals have MODIFY.
• Bundles: catalog names come from target-specific variables, with no default that points to prod.

variables:
  catalog:
    description: Target catalog (no default on purpose)
targets:
  dev:  { variables: { catalog: sales_dev } }
  prod: { variables: { catalog: sales_prod } }

• Jobs read the catalog from a parameter and fail fast if it does not match the workspace's environment tag.

#Prevent it next time

• Catalog-to-workspace bindings for every production catalog.
• No production write privileges for people or dev service principals.
• Alert on writes to production tables by principals outside an allow list (audit log query on a schedule).