Applies toData lakes (ADLS, S3, GCS), lakehouse tables (Delta, Iceberg), warehouses (Snowflake, Fabric, Synapse, Redshift), backups and exports
Inputs to the schedule
| Input | From |
|---|---|
| Legal and regulatory minimums/maximums | Legal, compliance, privacy officer |
| Business need for history | Data owners, analytics teams |
| Privacy principles (keep only as long as needed) | Privacy team |
| Cost | Platform team |
Retention schedule by data class
| Data class | Example | Raw (bronze) | Curated (silver/gold) | Notes |
|---|---|---|---|---|
| Personal data (customer) | Profiles, orders with PII | Short (to reprocess) | As per purpose | Erasure requests apply |
| Financial records | Invoices, ledger | Per regulation | Per regulation | Legal holds common |
| Operational telemetry | Clickstream, logs | Short | Aggregates kept longer | Aggregate, then delete detail |
| Reference data | Product catalog | Current + history as needed | Same | Low risk |
Fill in the actual periods with your legal team. They vary by country and industry.
Map retention to each layer
- Raw/bronze: keep only long enough to reprocess (e.g. weeks or months), especially for personal data.
- Silver/gold: keep per business purpose; aggregate or anonymise old detail.
- Table history (time travel, snapshots): deleted rows still exist in old versions until
VACUUMor snapshot expiry runs. - Backups, exports, copies in sandboxes: often forgotten; include them.
-- Delta: delete expired rows, then remove old files after the history window
DELETE FROM silver.web_events WHERE event_date < date_sub(current_date(), 395);
VACUUM silver.web_events RETAIN 168 HOURS;
-- Iceberg (Spark): expire old snapshots so deleted data files can be removed
CALL catalog.system.expire_snapshots(table => 'db.web_events', older_than => TIMESTAMP '2026-10-18 00:00:00');
Erasure requests and legal holds
| Need | Design |
|---|---|
| Right to erasure | Locate a person's data via catalog tags/keys; delete across layers; vacuum/expire history; record evidence |
| Legal hold | Flag data or containers on hold; deletion jobs skip them; release process recorded |
| Pseudonymisation | Keep personal identifiers in a separate, smaller table so erasure touches less data |
Automate and evidence
- Retention jobs run on schedules from a central configuration (table → period).
- Storage lifecycle rules for files outside tables (see the lake storage cost post).
- Logs of what was deleted, when, and by which job, kept for audit.
- Periodic check: oldest data per table vs its policy.
Caution
Deleting rows from a table doesn't remove them from storage until history cleanup runs. Erasure isn't complete until vacuum/snapshot expiry and backup retention have caught up.
Checklist
- Schedule agreed with legal, privacy and data owners
- Periods mapped per layer, history and backups
- Deletion jobs automated with central config
- Erasure and legal hold processes designed
- Evidence logged and compliance checked periodically