Sev1: Delta Lake pipelines failing with ConcurrentAppendException — fixing write conflicts

#Symptoms

• Jobs fail with ConcurrentAppendException: Files were added to the root of the table by a concurrent update, ConcurrentDeleteReadException or ConcurrentDeleteDeleteException.
• Failures started when a new job, backfill or OPTIMIZE schedule was added.

#1. Understand the conflict

Delta uses optimistic concurrency: each writer reads a snapshot, writes files, then tries to commit. The commit fails if another writer committed changes that overlap with what this one read.

Plain appends (INSERT INTO, .mode("append")) rarely conflict with each other.

#2. Find who else is writing

DESCRIBE HISTORY prod.sales.orders LIMIT 50;

Look at timestamp, operation, operationParameters, userName and job for writes around each failure. Usually you will see two MERGE jobs, or a MERGE overlapping with OPTIMIZE or DELETE (GDPR job, retention cleanup).

#3. Make predicates disjoint

If both writers touch different partitions, say so explicitly in the condition so Delta knows they don't conflict:

MERGE INTO prod.sales.orders t
USING updates_eu s
ON t.order_id = s.order_id
   AND t.region = 'EU'
   AND t.order_date >= '2026-10-01'
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Without the region and date predicates, the MERGE reads the whole table and conflicts with any other writer.

#4. Separate maintenance from writes

• Schedule OPTIMIZE and VACUUM in a window when the heavy writers are not running, or limit OPTIMIZE to older partitions:

OPTIMIZE prod.sales.orders WHERE order_date < current_date() - INTERVAL 2 DAYS;

• On Databricks, tables with deletion vectors and row-level concurrency (supported on recent runtimes) greatly reduce conflicts between MERGE, UPDATE and DELETE. Check your runtime's documentation before relying on it:

ALTER TABLE prod.sales.orders SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);

Warning

Enabling some table features upgrades the table protocol. Older readers or external engines may no longer read the table. Check every consumer first.

#5. Retry with backoff

Conflicts are safe to retry because nothing was committed. Wrap the write:

import time, random
from delta.exceptions import ConcurrentAppendException, ConcurrentDeleteReadException

for attempt in range(5):
    try:
        run_merge()
        break
    except (ConcurrentAppendException, ConcurrentDeleteReadException):
        if attempt == 4:
            raise
        time.sleep((2 ** attempt) + random.random())

#6. Serialize when necessary

If two jobs really must update the same rows, run them in sequence (one job with two tasks, or a job dependency) instead of in parallel.

#7. Recover today's runs

Rerun the failed jobs one at a time after the fix. Check DESCRIBE HISTORY to make sure each committed once, and look for duplicates if any job used non-idempotent appends.

#Prevent it next time

• One owner job per table for upserts; other teams write to staging tables.
• Partition or cluster on the columns used in MERGE conditions, and include them in predicates.
• Schedule maintenance away from load windows.
• Alert on retried and failed commits per table.