Auto Loader stream stopped after a new column: schema evolution modes and rescued data

#Symptom

An Auto Loader job that has run for months suddenly fails:

org.apache.spark.sql.catalyst.util.UnknownFieldException:
[UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE] Encountered unknown fields during parsing

Or the job keeps running, but new columns never appear, or columns fill up with nulls while _rescued_data grows.

#Why it stops: this is by design

With the default mode, Auto Loader stops when it finds a column that isn't in the stored schema. It has already recorded the new schema, so the next restart succeeds with the new column. You're expected to run the stream with automatic retries.

#The four modes

#Step 1: find out what changed

Look at the stored schema versions:

schema_path = "abfss://meta@stops.dfs.core.windows.net/autoloader/orders/schema"
display(dbutils.fs.ls(f"{schema_path}/_schemas/"))

print(dbutils.fs.head(f"{schema_path}/_schemas/1"))

Each file is a schema version. Compare the latest two.

Check the incoming file itself:

SELECT * FROM read_files(
  'abfss://landing@stops.dfs.core.windows.net/orders/2026/11/06/',
  format => 'json'
) LIMIT 5;

Ask the source team whether the change was intentional. A renamed column looks like one new column plus one that's always null, and that's a data problem, not just a schema change.

#Step 2: make restarts automatic

If you've chosen addNewColumns, plan for this failure rather than treating it as an incident:

df = (spark.readStream
      .format("cloudFiles")
      .option("cloudFiles.format", "json")
      .option("cloudFiles.schemaLocation", schema_path)
      .option("cloudFiles.schemaEvolutionMode", "addNewColumns")
      .load("abfss://landing@stops.dfs.core.windows.net/orders/"))

(df.writeStream
   .option("checkpointLocation", checkpoint)
   .option("mergeSchema", "true")
   .trigger(availableNow=True)
   .toTable("main.bronze.orders"))

mergeSchema lets the Delta target accept the new column. For continuous jobs in Lakeflow Jobs, configure a retry policy on the task, so a schema change produces one short restart:

{
  "max_retries": -1,
  "min_retry_interval_millis": 60000,
  "retry_on_timeout": false
}

Or use a job's continuous trigger, which restarts failed runs by itself. Alert on repeat failures, so a real problem doesn't loop silently. See job retries and repair.

#Step 3: control types with schema hints

For JSON and CSV, Auto Loader infers columns as strings by default, unless you enable cloudFiles.inferColumnTypes. A column inferred from only the first files can also end up wrong. Set the columns that matter yourself:

.option("cloudFiles.inferColumnTypes", "true")
.option("cloudFiles.schemaHints",
        "order_id BIGINT, amount DECIMAL(12,2), order_ts TIMESTAMP, customer.id STRING")

Hints apply to the named columns and leave inference for the others. They're also how you stop a numeric column being inferred as a string just because the first file's values were odd.

#Step 4: use rescued data

_rescued_data catches values that don't fit: wrong types, unexpected columns and, for some formats, differences in column case. It's a JSON string, with the file path included:

SELECT _rescued_data, count(*)
FROM main.bronze.orders
WHERE _rescued_data IS NOT NULL
  AND order_date >= current_date() - 1
GROUP BY 1
ORDER BY 2 DESC
LIMIT 20;

Get specific values:

SELECT order_id,
       _rescued_data:amount::string AS rescued_amount,
       _rescued_data:_file_path::string AS source_file
FROM main.bronze.orders
WHERE _rescued_data:amount IS NOT NULL;

Rescued data appears only when the schema is inferred, unless you set a rescue column yourself for a provided schema:

.option("rescuedDataColumn", "_rescued_data")

Warning

A healthy-looking stream can still be losing values into _rescued_data, leaving nulls in your real columns. Alert on the count of rows where it's not null. A rise usually means a source format change.

#Step 5: handle a type change

A column changing type, such as from integer to string, isn't handled by addNewColumns. The new values go to the rescue column, and the original column shows null.

To fix it:

1. Decide on a wider type that fits both forms. For example, use STRING in bronze, and cast in silver.
2. Add a schema hint for that column.
3. Change the target table's type. Delta allows some widening with type widening on supported runtimes; otherwise, rewrite the table:

CREATE OR REPLACE TABLE main.bronze.orders AS
SELECT * EXCEPT (amount), CAST(amount AS STRING) AS amount
FROM main.bronze.orders;

Caution

Rewriting a table changes its schema for all readers, and streaming readers from it may fail. A rewrite also needs a restart with care, because an existing checkpoint may not accept the new schema. Test on a copy, and see incompatible checkpoints first.

4. Backfill rows that went to rescue, using a one-off update from _rescued_data, or by reprocessing those files.

#Reprocess files safely

Deleting the schema location or the checkpoint is a last resort. Both have effects you can't undo:

A better way to reload specific files is a one-off batch job with read_files and a MERGE or replaceWhere. See stuck checkpoints if the checkpoint itself is damaged.

#Recommended setup by layer

from pyspark.sql.functions import col, current_timestamp

bronze = df.select("*",
                   col("_metadata.file_path").alias("source_file"),
                   current_timestamp().alias("ingest_ts"))

#Prevent it next time

• Agree data contracts with source teams, including notice of changes.
• Monitor rescued-row counts and schema version files, and alert on new versions.
• Use failOnNewColumns where unreviewed columns could expose sensitive data.
• Quarantine bad rows with expectations.
• Related: schema drift breaking loads and Delta schema mismatch errors.