Renaming or dropping a Delta column broke the streaming jobs: column mapping, schema tracking and safe rollout

#Symptom

You enabled column mapping and ran:

ALTER TABLE silver.customers RENAME COLUMN cust_nm TO customer_name;
ALTER TABLE silver.customers DROP COLUMN legacy_flag;

Minutes later, every downstream streaming job reading silver.customers fails. The error text varies by version, but it says the stream detected a non-additive schema change (rename or drop) and can't continue. Batch reads work fine.

Other things that may break at the same time:

• Change Data Feed readers spanning the change.
• External engines (older Delta readers, some Synapse/Athena/Trino versions) that don't support the column mapping table feature.

#Why

Without column mapping, Delta identifies columns by name in the Parquet files, so renames and drops would require rewriting every file. Column mapping (delta.columnMapping.mode = 'name') gives each column a stable ID and physical name, so RENAME and DROP are metadata-only.

A streaming source, however, started with a fixed schema. A rename means "the column it was reading no longer exists under that name". Silently continuing could corrupt the sink, so Delta stops the stream.

Enabling column mapping also upgrades the table protocol (reader version 2, writer version 5 or table features). Old readers can't read the table at all after that. See table features breaking external readers.

#Fix: restart the stream with schema tracking

Add a schemaTrackingLocation to the streaming read. Delta records the schema it's using there and can move the stream through rename/drop changes. It must be inside (or next to) the stream's checkpoint location:

checkpoint = "/Volumes/prod/chk/customers_to_gold"

(spark.readStream
   .option("schemaTrackingLocation", f"{checkpoint}/_schema")
   .table("silver.customers")
   .writeStream
   .option("checkpointLocation", checkpoint)
   .toTable("gold.customers"))

What happens next:

1. The first run with tracking may stop again after recording the new schema. That's by design; the stream restarts and continues with the new schema.
2. Delta then asks you to explicitly allow the non-additive change, by setting a Spark SQL conf. The error message prints the exact conf name and value to use (the names differ for rename only, drop only, or both, and can be scoped to a specific checkpoint and table version).

Example of the general form:

spark.conf.set("spark.databricks.delta.streaming.allowSourceColumnRenameAndDrop", "always")

Prefer the checkpoint-specific, version-specific form from the error message in production, so you're allowing this one change, not every future change.

Warning

Allowing the change means the stream continues with the new column names. If the sink table still has the old name, the write either fails or (with mergeSchema) adds a new column. Change the sink schema in the same release.

#Downstream checks before you resume

Find what reads the table, if you're on Unity Catalog:

SELECT DISTINCT entity_type, entity_id, source_table_full_name
FROM system.access.table_lineage
WHERE source_table_full_name = 'prod.silver.customers'
  AND event_date >= current_date() - 30;

#A safer rollout pattern

1. Enable column mapping on its own, in a release with no schema change. Confirm every reader still works.
2. For renames, consider add new column + backfill + switch readers + drop old column later. Each step is additive or isolated.
3. Add schemaTrackingLocation to all streaming reads of shared tables now, before you need it.
4. Do rename/drop changes in a window when you can restart the downstream streams.

#Prevent it next time

• Treat renames and drops on shared tables as breaking changes with a consumer list and a change ticket.
• Use lineage to build that consumer list automatically.
• Related: Delta schema mismatch errors, checkpoint incompatible after code change.