Transformation: plain Parquet/Hive tables to open table formats (Delta, Iceberg, UniForm)

#Why move off plain Parquet

Plain Parquet + Hive metastore gives no transactions (readers see half-written data), no reliable updates/deletes, slow partition listing and no time travel. Table formats add a transaction log/metadata layer on top of the same Parquet files.

#Delta or Iceberg?

If you're on Databricks but other engines read the data, Delta with UniForm gives Iceberg clients access without a second copy.

#1. Convert in place (no data rewrite)

Delta:

CONVERT TO DELTA parquet.`s3://lake/sales/transactions` PARTITIONED BY (sale_date DATE);
-- or for a Hive table
CONVERT TO DELTA hive_metastore.sales.transactions;

Iceberg (Spark procedures):

CALL spark_catalog.system.migrate('sales.transactions');      -- replaces the Hive table with an Iceberg table
CALL spark_catalog.system.snapshot('sales.transactions', 'sales.transactions_ice');  -- test copy, leaves source untouched

snapshot is ideal for testing: it creates an Iceberg table over the same files without changing the original.

Warning

After converting, stop any writer that writes plain Parquet files into the folder. Files written outside the table format are invisible to readers (or worse, collide with maintenance like VACUUM).

#2. Or rewrite (when you want a new layout)

CREATE TABLE main.sales.transactions
CLUSTER BY (sale_date, store_id)
TBLPROPERTIES ('delta.enableIcebergCompatV2' = 'true', 'delta.universalFormat.enabledFormats' = 'iceberg')
AS SELECT * FROM parquet.`s3://lake/sales/transactions`;

Rewriting fixes small files and lets you change partitioning/clustering at the same time.

#3. Change writers

• Spark jobs: .format("delta") / Iceberg catalog tables instead of .parquet(path).
• Replace "overwrite the partition folder" logic with MERGE, REPLACE WHERE or dynamic partition overwrite on the table.
• Ingestion tools (ADF, Glue, Kafka Connect sinks): switch to their Delta/Iceberg sink options.

#4. Change readers

Test each consumer before switching writers.

#5. Maintenance

• Delta: OPTIMIZE, VACUUM (with a retention that covers your time-travel and long-running reader needs), or predictive optimisation.
• Iceberg: rewrite_data_files, expire_snapshots, remove_orphan_files.

#Checklist

• Format chosen per consumer landscape
• Test copies via snapshot/clone; consumers verified
• Tables converted or rewritten; writers switched; plain Parquet writes stopped
• Maintenance jobs scheduled