Apache Iceberg with concurrent writers - a production checklist for catalogs, commit conflicts, isolation and metadata growth

#When to use this

Use it when an Iceberg table goes from one writer to many: a streaming append, a nightly MERGE, GDPR deletes and a compaction job, all against the same table. The usual failures are:

• CommitFailedException after retries are exhausted.
• ValidationException: Found conflicting files that can contain records matching ... from a MERGE or DELETE.
• Planning time that grows week by week.
• Many small files after a "successful" migration.

Work through each section before go-live. Every item is something that has caused a production incident.

#1. Catalog: the commit is only as safe as the catalog

An Iceberg commit is an atomic swap of the table's current metadata pointer. The catalog provides that atomicity.

• The catalog is hive, rest, glue, jdbc or nessie, not hadoop, for any table with more than one writer. The Hadoop catalog relies on atomic rename, which object stores such as S3 don't provide.
• Every engine (Spark, Trino, Flink, Snowflake or Databricks external access) points at the same catalog for writes. Two catalogs tracking the same table location is silent data loss: each one overwrites the other's commits.
• Hive Metastore catalogs: the metastore database and HMS service are sized for lock traffic (commits take an HMS lock), and HMS is highly available.
• REST catalogs: you know the server's own commit retry and rate limits.

spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.lake=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.lake.type=rest
spark.sql.catalog.lake.uri=https://catalog.internal/api/catalog
spark.sql.catalog.lake.warehouse=prod

#2. Commit retries match the concurrency

When two writers commit at once, one wins. The loser refreshes metadata, re-validates and retries. Appends almost always succeed on retry. Overwrites and row-level operations may fail validation instead (see section 3).

ALTER TABLE lake.sales.orders SET TBLPROPERTIES (
  'commit.retry.num-retries' = '10',
  'commit.retry.min-wait-ms' = '500'
);

• Retries are raised on hot tables. A retry is cheap for appends: only metadata is rewritten, not data.
• Streaming jobs commit no more often than needed (trigger interval of a minute or more, not seconds). Every commit is a new snapshot and a new metadata file, and it competes with every other writer.

#3. Isolation levels and conflict detection

Row-level operations validate that no conflicting change landed since they read the table:

serializable fails a MERGE if another writer added files that could contain matching rows. snapshot only fails on conflicting deletes. It allows more concurrency, but a MERGE can then miss rows appended concurrently.

• The MERGE condition includes the partition column, so validation is scoped to the partitions it touches. A MERGE that matches on order_id alone conflicts with every append to the table:

MERGE INTO lake.sales.orders t
USING updates s
ON t.order_date = s.order_date AND t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

• Writers that touch disjoint partitions are scheduled so they actually stay disjoint. A late-data append to yesterday's partition conflicts with a MERGE on yesterday.
• Jobs that hit ValidationException retry the whole job (re-read and recompute). Iceberg can't retry a failed validation for you.
• snapshot isolation is a documented decision for that table, not a quick fix for a conflict.

#4. Copy-on-write or merge-on-read

• Frequent small MERGE/DELETE on large partitions use merge-on-read. Copy-on-write rewrites whole data files for a handful of changed rows, so commits are long and conflict windows are wide.
• Every engine that reads the table supports v2 delete files. Old readers ignore them or fail, which shows deleted rows.
• Merge-on-read tables have scheduled delete-file compaction, or read performance degrades:

CALL lake.system.rewrite_position_delete_files(table => 'sales.orders');
CALL lake.system.rewrite_data_files(
  table => 'sales.orders',
  options => map('delete-file-threshold', '5'));

#5. Write distribution and file size

• write.distribution-mode is set explicitly on partitioned tables (hash for most, range when there is a sort order). Spark's default depends on the Iceberg and Spark versions. Without clustering by partition, each task writes a file into every partition it sees, which is the small-files explosion.

ALTER TABLE lake.sales.orders SET TBLPROPERTIES (
  'write.distribution-mode' = 'hash',
  'write.target-file-size-bytes' = '536870912'
);

• Streaming appends, which can't be large, have scheduled rewrite_data_files with a where clause limited to recent partitions.

#6. Compaction doesn't fight the writers

Compaction is a writer too. A big rewrite_data_files that loses a commit race throws away hours of work.

• Compaction uses partial progress, so it commits in groups instead of one big commit at the end:

CALL lake.system.rewrite_data_files(
  table => 'sales.orders',
  where => 'order_date >= current_date() - INTERVAL 7 DAYS',
  options => map(
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '10',
    'max-concurrent-file-group-rewrites', '8'));

• Compaction of partitions that a MERGE is rewriting is scheduled outside that MERGE's window.
• Only one compaction job runs per table at a time (enforce it in the orchestrator).

#7. Metadata growth

Each commit writes a new metadata.json. Each snapshot keeps manifests alive. With high-frequency commits, planning slows and the metadata directory grows.

• Old metadata files are cleaned up on commit:

ALTER TABLE lake.sales.orders SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled' = 'true',
  'write.metadata.previous-versions-max' = '50'
);

• Snapshot expiry runs on a schedule (history.expire.max-snapshot-age-ms defaults to 5 days, but nothing expires until you run the procedure):

CALL lake.system.expire_snapshots(
  table => 'sales.orders',
  older_than => TIMESTAMP '2026-11-02 00:00:00',
  retain_last => 50);
CALL lake.system.rewrite_manifests('sales.orders');

• Health is monitored from metadata tables:

SELECT count(*) AS snapshots FROM lake.sales.orders.snapshots;
SELECT count(*) AS manifests, sum(added_data_files_count) FROM lake.sales.orders.manifests;
SELECT committed_at, operation, summary['added-data-files'] AS added
FROM lake.sales.orders.snapshots ORDER BY committed_at DESC LIMIT 20;

Caution

remove_orphan_files with a short older_than deletes files that an in-flight writer has written but not yet committed. That writer's commit then references missing files, and the table is corrupt. Keep older_than at least as long as your longest-running write job (days, not minutes).

#8. Audit before publish (optional)

For tables that downstream teams must never see half-written, use write-audit-publish on a branch:

ALTER TABLE lake.sales.orders SET TBLPROPERTIES ('write.wap.enabled' = 'true');
ALTER TABLE lake.sales.orders CREATE BRANCH audit_20261109;

spark.conf.set("spark.wap.branch", "audit_20261109")
# run the load, then data quality checks against the branch
spark.conf.unset("spark.wap.branch")
spark.sql("CALL lake.system.fast_forward('sales.orders', 'main', 'audit_20261109')")

• Publish (fast_forward) is a single small commit, so it rarely conflicts.

#9. Observability and ownership

• Commit failures and retry counts from writer logs are collected and alerted on (search for CommitFailedException and ValidationException).
• Each table has one owner who approves changes to properties, partition spec and maintenance schedules.
• Snapshot summaries carry the job name, so you can tell from .snapshots which writer did what:

(df.writeTo("lake.sales.orders")
   .option("snapshot-property.job", "orders_nightly_merge")
   .append())

#Prevent it next time

Put these table properties in the table DDL template, not in the tribal knowledge of whoever created the first table. Review new multi-writer tables against this checklist before the second writer is switched on.