Delta Lake ConcurrentAppendException and other write conflicts: find the clash and stop it

#The symptom

A job that ran fine for months fails with one of these:

io.delta.exceptions.ConcurrentAppendException: Files were added to the root of the table by a concurrent update. Please try the operation again.
io.delta.exceptions.ConcurrentDeleteReadException: This transaction attempted to read one or more files that were deleted ...
io.delta.exceptions.MetadataChangedException: The metadata of the Delta table has been changed by a concurrent update ...

Newer runtimes also show an error class such as DELTA_CONCURRENT_APPEND. Names and message text vary by version; check your build's output.

Delta uses optimistic concurrency. Each writer reads a snapshot, writes new files, and then tries to commit the next table version. If another writer committed first and touched files your transaction depended on, your commit is rejected. Nothing is corrupted. The failed transaction simply did not commit. The task is to find the other writer and remove the overlap.

#Which exception means what

#Checks

#1. Find the commit that beat you

Every committed write is in the table history. Look at the versions around the failure time:

SELECT version, timestamp, operation, operationParameters,
       isolationLevel, isBlindAppend, userName, job, clusterId
FROM (DESCRIBE HISTORY main.sales.orders LIMIT 50)
WHERE timestamp BETWEEN '2026-10-23T02:00:00' AND '2026-10-23T02:30:00'
ORDER BY version;

The job and clusterId columns tell you which job or cluster made the competing commit. isBlindAppend = true means a plain INSERT that read nothing, which cannot cause a conflict on its own side but can trigger one on a MERGE that reads the same partition.

#2. Check how the table is laid out

DESCRIBE DETAIL main.sales.orders;   -- partitionColumns, clusteringColumns, numFiles
SHOW TBLPROPERTIES main.sales.orders;

You are looking for:

• partitionColumns: if the table is partitioned by date, a MERGE without a date predicate on the target reads every partition, so any concurrent append anywhere conflicts.
• delta.enableDeletionVectors: required for row-level concurrency.
• delta.isolationLevel: WriteSerializable is the default; Serializable conflicts more often.

#3. Check the operation's predicate

Look at the failing MERGE, UPDATE or DELETE. If the condition only joins on keys, Delta has to assume it read the whole table:

-- Reads every partition; conflicts with any concurrent write
MERGE INTO main.sales.orders t
USING staged_orders s
ON s.order_id = t.order_id
...

#Fixes

#Make the data each writer touches explicit

For partitioned tables, put literal partition values on the target side of the condition so Delta knows which partitions the transaction read:

MERGE INTO main.sales.orders t
USING staged_orders s
ON  s.order_id = t.order_id
AND t.order_date = '2026-10-22'
AND t.region = 'emea'
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Two jobs running this for different dates or regions now commit independently. The same applies to UPDATE and DELETE: DELETE FROM t WHERE region = 'emea' AND ... is safe alongside a writer on region = 'apac' only if the table is partitioned by region.

#Turn on row-level concurrency (Databricks)

On Databricks Runtime 14.3 LTS and above, row-level concurrency is on automatically when the table is not partitioned and has deletion vectors enabled. Tables with liquid clustering get it too. With it, conflicts are detected per row instead of per file:

ALTER TABLE main.sales.orders SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);

Warning

Enabling deletion vectors is a protocol change. It fails every in-flight write with MetadataChangedException, and older readers or external engines that don't support deletion vectors can no longer read the table. Pause writers, check your downstream readers, then make the change.

Row-level detection falls back to file-level for conditions on structs, arrays or maps, non-deterministic expressions and subqueries. Behaviour on 13.3 LTS and 14.2 differs; check the docs for your runtime.

#Stop maintenance clashing with pipelines

OPTIMIZE ... ZORDER BY rewrites files that MERGE jobs are reading. Either move to liquid clustering, or run OPTIMIZE from the same job after the write, not from a separate schedule that overlaps it. Never run two OPTIMIZE jobs on the same table.

#Serialise writers that really overlap

If two jobs must update the same rows, concurrency control cannot help. Put them in one job with task dependencies, or stage both into a single MERGE.

#Retry, with a limit

A short retry is reasonable for occasional conflicts. It is not a fix for jobs that conflict every run.

import random, time
from delta.exceptions import ConcurrentAppendException, ConcurrentDeleteReadException

def merge_with_retry(run_merge, attempts=4):
    for i in range(attempts):
        try:
            return run_merge()
        except (ConcurrentAppendException, ConcurrentDeleteReadException):
            if i == attempts - 1:
                raise
            time.sleep(2 ** i + random.random())

Caution

Only retry operations that are idempotent. A MERGE keyed on a business key is safe to rerun. An INSERT INTO that partially succeeded elsewhere in the job, or a non-idempotent UPDATE like SET qty = qty + 1, will double-apply.

#Fix the other exceptions

• ConcurrentTransactionException: in the Jobs UI set Maximum concurrent runs to 1 for the streaming job and check that no two queries share a checkpointLocation.
• MetadataChangedException: find the SET TBLPROPERTIES, ADD COLUMNS or mergeSchema write in DESCRIBE HISTORY. Move schema changes into a release step.
• ProtocolChangedException: create the table with explicit DDL in a setup task before parallel writers start.

#Prevent it next time

• One writer per partition, or per table, wherever you can design it that way.
• New tables: use liquid clustering or no partitioning, with deletion vectors, on 14.3 LTS or later.
• Always include target-side partition predicates in MERGE on partitioned tables.
• Set Maximum concurrent runs = 1 on jobs that write the same table.
• Keep DDL and table-feature changes out of the pipeline schedule.
• Alert on repeated conflicts: if the same table shows Concurrent* errors in job failure notifications more than once a week, treat it as a design problem, not a retry problem.