Spark overwrite wiped the whole table: dynamic partition overwrite, replaceWhere and recovery

#Symptom

A daily job reloads one date. Afterwards, the table holds only that date, and everything else is gone. The job reported success, with no errors.

This code looks harmless, and it's a classic cause:

daily_df.write.mode("overwrite").partitionBy("order_date").parquet("abfss://lake@st.dfs.core.windows.net/orders")

#Why it happens

Spark's default is static overwrite (spark.sql.sources.partitionOverwriteMode=static). In this mode, mode("overwrite") replaces everything under the target that the write doesn't explicitly exclude. Partitioning the output by order_date doesn't limit what gets deleted.

#Stop first: limit the damage

1. Pause the job and any downstream job that reads this table.
2. For Delta, don't run VACUUM, and hold any scheduled maintenance. It may remove the files you need.
3. For cloud storage, pause lifecycle rules that clean up old versions.
4. Note the time of the bad write and the job run ID.

Caution

Recovery windows are short. VACUUM on Delta, a lifecycle rule on storage or a soft-delete expiry can make this permanent. Stop those before you investigate further.

#Recover a Delta table

Delta keeps prior versions, so this is usually straightforward:

DESCRIBE HISTORY orders LIMIT 20;

Look for the WRITE operation with mode: Overwrite, and check operationMetrics for the number of files and rows. Then look at the version before it:

SELECT COUNT(*), MIN(order_date), MAX(order_date) FROM orders VERSION AS OF 41;

If it's right, restore it:

RESTORE TABLE orders TO VERSION AS OF 41;

A restore is itself a new table version, so you can undo it. If data has been written since the bad overwrite, restoring also removes that newer data. In that case, take only the affected partitions back from the old version:

INSERT INTO orders
SELECT * FROM orders VERSION AS OF 41
WHERE order_date < '2026-11-06';

Check the counts afterwards. See Delta backup and recovery for more.

#Recover a plain Parquet table

Plain Parquet has no history of its own. Your options, in order:

On Azure Data Lake with soft delete enabled (hierarchical namespace accounts have their own support; check your account):

az storage fs file list --account-name st --file-system lake --path orders --auth-mode login -o table
az storage blob list --account-name st --container-name lake --prefix orders/ \
  --include d --auth-mode login --query "[?deleted].name" -o tsv

On S3 with versioning, see recovering deleted objects. After restoring, repair the metastore so it sees the partitions:

MSCK REPAIR TABLE orders;

#The fix: choose the right overwrite

#Option 1: dynamic partition overwrite

Replace only the partitions present in the data:

spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")

(daily_df.write
    .mode("overwrite")
    .insertInto("orders"))

Or set it just for this write:

(daily_df.write
    .mode("overwrite")
    .option("partitionOverwriteMode", "dynamic")
    .partitionBy("order_date")
    .parquet(path))

Note the behavior of insertInto: it matches columns by position, not by name, so select columns in the table's order. Delta supports dynamic overwrite on recent versions, and there's an option conflict: don't combine it with replaceWhere.

#Option 2: Delta replaceWhere

This is explicit, and it works on any column. It's the safest choice for Delta:

(daily_df.write
    .format("delta")
    .mode("overwrite")
    .option("replaceWhere", "order_date = '2026-11-06'")
    .saveAsTable("orders"))

Delta also checks that incoming rows match the predicate, and fails the write if they don't, which guards against loading the wrong date. Details vary by Delta version.

#Option 3: name the partition in SQL

INSERT OVERWRITE TABLE orders PARTITION (order_date = '2026-11-06')
SELECT order_id, customer_id, amount FROM staging_orders;

Which one to use:

Warning

Dynamic overwrite has a quiet failure of its own. If an upstream problem gives you an empty DataFrame, nothing gets replaced, so stale data stays and the job still succeeds. A partial DataFrame also replaces a full partition with partial data. Add checks.

#Add guards to the job

def safe_reload(df, table, run_date, min_rows):
    rows = df.count()
    if rows < min_rows:
        raise ValueError(f"{table}: {rows} rows is below the minimum {min_rows}")

    bad = df.filter(df.order_date != run_date).count()
    if bad:
        raise ValueError(f"{table}: {bad} rows are outside {run_date}")

    (df.write.format("delta").mode("overwrite")
        .option("replaceWhere", f"order_date = '{run_date}'")
        .saveAsTable(table))

    written = spark.table(table).filter(f"order_date = '{run_date}'").count()
    if written != rows:
        raise ValueError(f"{table}: wrote {written}, expected {rows}")

Base the minimum on recent history, for example 50% of the 30-day median.

For a cluster-wide safety default on self-managed Spark:

spark.sql.sources.partitionOverwriteMode  dynamic

Test this before you apply it broadly, because any job that truly expects a full overwrite will behave differently.

Iceberg tables: dynamic overwrite is supported, and overwritePartitions() in the DataFrameWriterV2 API makes the intent clear. Use DF.writeTo("t").overwritePartitions().

#Prevent it next time

• Ban bare mode("overwrite") on partitioned tables in code review, and catch it with a lint check.
• Keep Delta retention long enough to notice a bad job, and turn on storage soft delete.
• Alert on sudden row-count drops. See data quality checks.
• Make all reloads idempotent, so a rerun gives the same result.