Applies toDelta Lake, Apache Iceberg, Apache Hudi; engines such as Spark, Databricks, Snowflake, Trino, Athena, BigQuery, Fabric, Flink; catalogs such as Unity Catalog, AWS Glue, Polaris/REST catalogs
What all three provide
ACID transactions on object storage, schema evolution, time travel, and metadata that lets engines skip files. The differences are ecosystem, catalog integration and operational features.
Decide by engines and catalog
| Question | Why |
|---|---|
| Which engines write the tables? | Writers need full support; concurrent writers need compatible commit protocols |
| Which engines read them? | Readers may support only some versions/features |
| Which catalog manages them? | Catalog choice ties into governance, permissions and discovery |
| Streaming upserts at high frequency? | Some formats/engines are stronger at CDC upserts |
| Ecosystem tendency | Format |
|---|---|
| Databricks-centric, Fabric/OneLake | Delta Lake (with Iceberg compatibility features for other readers) |
| Multi-engine on AWS (Athena, EMR, Glue, Redshift), Snowflake, Trino, BigQuery | Iceberg is widely supported |
| Upsert-heavy CDC pipelines with record-level indexing needs | Hudi (or Delta/Iceberg with merge-on-read features) |
Support changes quickly. Check each engine's current read/write support and feature level (deletion vectors, row-level deletes, v2/v3 specs).
Interoperability instead of copies
- Delta with UniForm / Iceberg compatibility: Delta tables that also publish Iceberg metadata so Iceberg readers can query them.
- Iceberg REST catalogs: let several engines share one catalog for Iceberg tables.
- Shortcuts / external tables: point other platforms at the same files rather than copying.
- XTable-style metadata translation: generate metadata for another format from existing tables.
Each has limits (features supported, write paths, catalog sync). Test with your engines.
Operational features to compare
| Feature | Questions |
|---|---|
| Compaction and clustering | Built-in automatic maintenance on your platform? |
| Deletes/updates | Copy-on-write vs merge-on-read, deletion vectors, impact on readers |
| Snapshot expiry / vacuum | How history is cleaned up, and by whom |
| Concurrency | How conflicts are detected (see the Delta concurrency post) |
| Partition evolution | Change partitioning without rewriting (Iceberg strength; liquid clustering in Delta) |
Recommendations
- Pick one primary format per platform; avoid a mix decided pipeline by pipeline.
- Choose the catalog with the same care. It's where permissions and discovery live.
- Use interoperability features for cross-engine reads; keep writes on the engine that owns the table.
- Document supported reader/writer engines per table class.
Warning
Writing the same table from two different engines with partial format support is a common source of corruption-like issues (unreadable versions, lost commits). Designate one writer engine per table unless both are verified compatible.
Checklist
- Reader and writer engines listed per data domain
- Catalog strategy chosen
- Primary format decided
- Interoperability approach tested
- Maintenance (compaction, expiry) ownership assigned
- Writer engine per table documented