Apache Iceberg hidden partitioning and partition evolution: change the layout without rewriting the table

In Hive-style tables the partition is a directory and a column. Change your mind about the layout and you rewrite everything, and every query must filter on the partition column explicitly. Iceberg separates the two: partitions are derived from data columns by transforms, and the spec can change over time.

#Transforms

CREATE TABLE lake.events (
  event_id    BIGINT,
  customer_id BIGINT,
  event_ts    TIMESTAMP,
  type        STRING
) USING iceberg
PARTITIONED BY (days(event_ts), bucket(16, customer_id));

#Hidden partitioning

Queries filter on the real column; Iceberg maps that to partitions:

SELECT count(*) FROM lake.events
WHERE event_ts >= TIMESTAMP '2026-10-28 00:00:00' AND event_ts < TIMESTAMP '2026-10-29 00:00:00';

No event_date column, no risk of a user filtering on the timestamp and accidentally scanning everything. Writers do not need to compute partition values either.

#Evolve the spec

Traffic grew and daily partitions are now too big:

ALTER TABLE lake.events REPLACE PARTITION FIELD days(event_ts) WITH hours(event_ts);

Add or drop fields:

ALTER TABLE lake.events ADD PARTITION FIELD truncate(2, type);
ALTER TABLE lake.events DROP PARTITION FIELD bucket(16, customer_id);

These are metadata-only changes. Existing files keep the old spec; new writes use the new one. Each data file records which spec it was written with, and planning handles both.

Warning

These ALTER ... PARTITION FIELD statements need the Iceberg Spark SQL extensions (spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions). Without them Spark returns a parse error.

#Check pruning works

Inspect specs and files:

SELECT * FROM lake.events.partitions ORDER BY partition LIMIT 20;
SELECT spec_id, count(*) AS files, sum(file_size_in_bytes)/1e9 AS gb
FROM lake.events.files GROUP BY spec_id;

In Spark, the scan node of EXPLAIN shows pushed filters; Iceberg's scan metrics (or Spark UI "number of files read") confirm how many files were skipped. In Trino, EXPLAIN ANALYZE shows the input files and bytes.

#Rewrite old data (optional)

Old files under the daily spec still prune by day, which may be fine. If you want everything on the new layout:

CALL lake.system.rewrite_data_files(
  table => 'lake.events',
  where => 'event_ts >= TIMESTAMP ''2026-01-01 00:00:00''',
  options => map('target-file-size-bytes', '536870912', 'rewrite-all', 'true')
);

Run it in time windows to control cost, then expire old snapshots so the replaced files can be removed:

CALL lake.system.expire_snapshots(table => 'lake.events', older_than => TIMESTAMP '2026-10-22 00:00:00', retain_last => 20);
CALL lake.system.remove_orphan_files(table => 'lake.events', older_than => TIMESTAMP '2026-10-22 00:00:00');

Caution

expire_snapshots ends time travel to the expired versions, and remove_orphan_files with a too-recent older_than can delete files from in-flight writes. Keep a margin of days, not minutes.

#Design guidance

Sort orders complement partitions: ALTER TABLE lake.events WRITE ORDERED BY customer_id, event_ts; improves min/max skipping inside partitions.

#Engine notes

• Most engines read evolved specs correctly. Very old readers may not; test before evolving tables shared with external tools.
• Partition evolution is different from schema evolution (columns), which Iceberg also supports by field ID.

#Prevent it next time

• Start with a coarse spec and evolve as data grows; Iceberg makes that cheap.
• Monitor average file size per partition; rewrite when it drifts far from target.
• Schedule snapshot expiry and orphan cleanup as part of table maintenance.