Home / Data Platform / Governance
Platform-wide design: governance, data mesh, contracts, environments, observability, cost and reviews. 11 posts.
When a report shows stale or missing data, work backwards from the report through the semantic model, warehouse tables, pipelines and source. Find the first layer where the data stops, fix it, backfill and tell users.
A neutral design guide for picking the core analytics platform. Start from workloads, skills and existing estate rather than feature lists, compare the platforms on the questions that decide operations and cost, run a proof of value with your own data, and plan for coexistence because most organisations end up with more than one.
Shared warehouses, clusters and capacities make it hard to see who drives cost. Choose allocation keys per platform, tag or separate compute by team, split shared costs fairly, start with showback before any chargeback, and use the numbers to drive optimisation rather than arguments.
A data contract is a written, versioned agreement on what a data product provides. Define the schema, semantics, quality and freshness guarantees, owners and change process, enforce what you can in CI and at runtime, and introduce contracts first where breakage hurts most.
Keeping everything forever raises cost and privacy risk; deleting too early breaks audits. Build a retention schedule per data class with legal and business input, map it to each layer, implement deletion that also covers table history and backups, support legal holds and erasure requests, and prove it with evidence.
Data platforms need separate environments like any software, but data makes it harder. Choose how environments are isolated, decide what data each one gets (synthetic, masked, sampled or read-only production), keep code promotion automated and identical, and stop sandboxes from becoming unmanaged copies of production.
A structured review across reliability, security, cost, performance and operations finds the risks that build up in a data platform over time. Scope it, gather evidence before the workshop, use a question set tailored to data workloads, rate risks, and turn findings into a short, owned improvement plan.
Users don't care that a job succeeded; they care that the data is fresh and correct by the time they need it. Define data SLOs per critical dataset, collect pipeline, platform and data signals in one place, alert on SLO risk rather than every task failure, and review SLO performance with consumers.
A practical, tool-level path from one overloaded central data team to domain teams that own data products. Define what a data product is, set up the catalog and permissions per domain, give domains a self-serve platform and keep shared governance.
Separate dev, test and prod, put pipelines, notebooks, SQL models and infrastructure in Git, run automated tests on pull requests and promote releases with parameterised configs.
The minimum governance setup that makes a data platform supportable, covering scanning and cataloguing, ownership, sensitive data classification, lineage, access request workflow and audit.
No posts in this category yet.