1. Measure by zone and dataset
Use storage inventory reports (Azure Blob inventory, S3 Inventory) and aggregate by path prefix:
-- e.g. DuckDB / Spark SQL over an inventory export
SELECT split_part(name, '/', 1) AS zone, split_part(name, '/', 2) AS dataset,
ROUND(SUM(content_length) / 1e12, 2) AS tb, COUNT(*) AS files,
MIN(last_modified) AS oldest
FROM inventory
GROUP BY 1, 2 ORDER BY tb DESC LIMIT 30;
Typical findings: raw zone holds years of uncompressed CSV/JSON, bronze duplicates raw, Delta tables carry many old file versions, and tmp/ or sandbox/ paths are huge.
2. Retention per zone
| Zone | Purpose | Example retention |
|---|---|---|
| Landing / raw | As-received files for replay and audit | 30–90 days hot, then cool/archive or delete if bronze is a faithful copy |
| Bronze | Raw data as tables (append-only) | Long, as the system of record for reprocessing |
| Silver | Cleaned, conformed | Rebuildable from bronze; history per business need |
| Gold | Serving/marts | Rebuildable |
| Sandbox / tmp | Experiments | 30 days, auto-delete |
Agree with data owners and compliance; record in the data catalog.
3. Formats and compression
| From | To | Effect |
|---|---|---|
| CSV/JSON in raw kept forever | Parquet/Delta in bronze, raw expires | Often several times smaller, faster to query |
| Snappy-compressed Parquet | zstd (where engines support it) | Smaller files, similar speed |
| Millions of tiny files | Compacted files of 128 MB–1 GB | Fewer transactions, faster queries (see the small-files post) |
4. Storage tiers
Lifecycle rules on the raw/landing prefix and on archives (see the Azure Blob and S3 lifecycle posts). Avoid tiering table data files (Delta/Iceberg) to archive tiers. Queries and maintenance would fail or become expensive. Cool tiers for rarely queried table data need careful testing of read costs.
5. Table history
Delta/Iceberg keep old data files for time travel until vacuumed/expired:
-- Delta: keep 7 days of history (default threshold), then remove unreferenced files
VACUUM bronze.orders RETAIN 168 HOURS;
-- Iceberg (Spark procedure)
CALL catalog.system.expire_snapshots(table => 'bronze.orders', older_than => TIMESTAMP '2026-10-18 00:00:00');
CALL catalog.system.remove_orphan_files(table => 'bronze.orders');
Schedule maintenance (or use managed/predictive optimization). Tables with frequent overwrites and no vacuum can hold many times their live size.
6. Orphaned and duplicate data
- Paths not registered as any table in the catalog, with no reads in 90 days (storage logs/access tracking).
- Extracts copied to "temp" for one-off analysis.
- Duplicate copies of the same dataset in several zones or teams. Share instead (views, Delta Sharing, shortcuts).
7. Keep it under control
- Storage cost by zone/dataset monthly, with owners.
- Default lifecycle rules for
tmp/andsandbox/created with each new workspace/container. - Ingestion frameworks that write compressed columnar formats by default.
Caution
Deleting raw files is irreversible. Before shortening raw retention, confirm bronze holds everything needed to reprocess (including late corrections), and that no compliance rule requires the original files.
Checklist
- Size by zone and dataset from inventory
- Retention agreed per zone
- Raw formats converted; compaction in place
- Lifecycle tiers on raw/archives, not table files
- VACUUM/expire snapshots scheduled
- Orphans and duplicates removed; monthly review