Home / Data Platform / Lakehouse
Lakehouse storage: Delta Lake, Iceberg, Parquet, medallion design, table maintenance and recovery. 13 posts.
How Iceberg partition transforms work, why queries do not need partition columns, how to evolve a table from daily to hourly or add bucketing without rewriting history, and how to check pruning and compact old layouts when you want to.
What each common data file format is good and bad at, why Parquet is the default for analytics, where Avro and JSON still fit, and how to inspect and convert files with Python, Spark and DuckDB.
A first hands-on tour of Delta Lake. Create a Delta table, append and upsert data with MERGE, read the transaction history, travel back to an older version and restore it, with SQL and PySpark side by side.
Iceberg tables slow down and grow in storage without maintenance. Schedule rewrite_data_files for small files, expire_snapshots for old versions, rewrite_manifests for planning speed and remove_orphan_files for leftovers.
Open table formats give the lake ACID tables, time travel and schema evolution, but engines support them differently. Decide by which engines must read and write the data, the catalog you'll use and the operational features you need, and use interoperability options instead of copying data between formats.
Data lakes grow because every layer keeps everything forever - raw files, bronze copies, old table versions and test extracts. Measure size by zone and dataset, agree retention per zone, convert raw formats to compressed columnar files, tier cold data, vacuum table history and remove orphaned paths.
Revenue doubled overnight because a failed load was re-run on top of a partial one. Find the duplicates, remove them safely in Delta Lake, Snowflake or SQL, and redesign the load with MERGE or partition overwrite so re-runs are always safe.
Recover when a source system renames, retypes or drops columns and pipelines fail or, worse, load nulls silently. Detect the change, quarantine bad batches, adapt the load safely and put schema checks and contracts in place.
Upgrade folders of Parquet files registered as Hive tables to a transactional table format. Choose Delta or Iceberg (or both via UniForm), convert in place or rewrite, and change writers and readers without breaking consumers.
Two jobs writing the same Delta table and one fails with ConcurrentAppendException or a sibling. Find the other writer, then fix it with predicates, row-level concurrency or scheduling.
Recover from bad writes, accidental deletes and storage loss in a Delta lakehouse. Covers time travel and RESTORE, what VACUUM does to your recovery window, deep clone backups, and storage soft delete and versioning.
Keep Delta tables fast and storage under control by compacting small files, choosing between Z-ORDER and liquid clustering, and running VACUUM safely.
Define bronze, silver and gold layers, the storage account/bucket layout per environment, table naming, file formats and ownership rules before the first pipeline lands data.
No posts in this category yet.