Apache Iceberg table maintenance - expiring snapshots, compacting files and removing orphans with Spark

#Why maintenance matters

Every write to an Iceberg table creates a new snapshot, new metadata files and usually new data files. Over time you get:

Iceberg doesn't clean any of this up on its own. You run maintenance procedures, usually from Spark.

#Setup

The examples use a Spark catalog named lake and the table lake.sales.orders:

spark-sql \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.lake=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.lake.type=hive

The CALL procedures need the Iceberg Spark extensions.

#1. Look before you clean

-- snapshots
SELECT committed_at, snapshot_id, operation, summary['added-data-files'] AS added
FROM lake.sales.orders.snapshots ORDER BY committed_at DESC LIMIT 20;

-- file count and average size per partition
SELECT partition, count(*) AS files, round(avg(file_size_in_bytes)/1024/1024,1) AS avg_mb
FROM lake.sales.orders.files GROUP BY partition ORDER BY files DESC LIMIT 20;

-- manifests
SELECT count(*) AS manifests FROM lake.sales.orders.manifests;

Partitions with hundreds of files averaging a few MB are candidates for compaction.

#2. Compact data files

CALL lake.system.rewrite_data_files(
  table => 'sales.orders',
  strategy => 'binpack',
  where => 'order_date >= date ''2026-10-01''',
  options => map('target-file-size-bytes','536870912','min-input-files','5','partial-progress.enabled','true')
);

• binpack merges small files. sort (with sort_order) also clusters data for better file skipping, but it costs more.
• Filter with where to compact recent partitions only. Old partitions that are already compacted don't need it again.
• partial-progress.enabled commits in groups, so a long job doesn't lose all its work if it fails and conflicts less with concurrent writers.

Tip

Run compaction for streaming tables at least daily, or after each batch of micro-batches. Compacting a partition once it's "closed" (yesterday's date) avoids conflicts with the live writer.

#3. Expire old snapshots

CALL lake.system.expire_snapshots(
  table => 'sales.orders',
  older_than => TIMESTAMP '2026-10-19 00:00:00',
  retain_last => 50
);

This removes snapshots older than the timestamp (always keeping at least retain_last) and deletes the data and metadata files that only those snapshots used.

Warning

After expiry you can't time travel or roll back to those snapshots. Agree the retention with the data owners. Seven days is a common default; regulated data may need longer. Tags and branches can protect specific snapshots.

You can also set the retention as table properties, so jobs don't need to hard-code it:

ALTER TABLE lake.sales.orders SET TBLPROPERTIES (
  'history.expire.max-snapshot-age-ms'='604800000',
  'history.expire.min-snapshots-to-keep'='50'
);

Also limit old metadata JSON files:

ALTER TABLE lake.sales.orders SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled'='true',
  'write.metadata.previous-versions-max'='100'
);

#4. Rewrite manifests

CALL lake.system.rewrite_manifests('sales.orders');

This helps when planning is slow and the table has many small manifests, which is typical after many small appends.

#5. Remove orphan files

CALL lake.system.remove_orphan_files(
  table => 'sales.orders',
  older_than => TIMESTAMP '2026-10-23 00:00:00',
  dry_run => true
);

Run with dry_run => true first and check the list.

Caution

Never use a short older_than here. Files from writes that are still in progress look like orphans until they're committed. Keep at least a few days' margin. Also make sure no other table or tool shares the same location.

#Suggested schedule

Run maintenance as a separate job with its own cluster, outside the main writer's schedule. Watch for commit conflicts: maintenance and writers both commit, and Iceberg retries, but heavy contention can make both fail.

#Managed alternatives

Some catalogs and services can run these operations for you (for example AWS Glue table optimizers for Iceberg tables in the Glue Data Catalog, or other vendors' managed compaction). If you use one, don't also run the same jobs yourself.

#Checklist

• File sizes and snapshot counts measured per table
• Daily compaction of recent partitions
• Snapshot retention agreed and set as table properties
• Old metadata files limited
• Orphan cleanup with dry run and a safe margin
• Maintenance scheduled away from heavy writes