Transformation: from a central data team bottleneck to domain-owned data products

#The symptom

Every new report waits in the central team's backlog; source teams change systems without telling anyone; the central team can't understand every business domain. "Data mesh" is one answer — but it fails when it's only a slide. Here's the operational version.

#1. Define a data product (one page)

A data product is a dataset a domain team publishes for others, with:

• An owner (team and named person) and support channel.
• A documented schema and meaning of each column.
• Freshness and quality SLOs, measured.
• Access through the shared catalog, with a request process.
• Versioning: breaking changes get a new version and a deprecation period.

#2. Map domains to platform boundaries

Unity Catalog example:

CREATE CATALOG sales MANAGED LOCATION 'abfss://sales@stlake.dfs.core.windows.net/';
GRANT USE CATALOG, CREATE SCHEMA ON CATALOG sales TO `grp-sales-data-engineers`;
CREATE SCHEMA sales.products;     -- published data products
CREATE SCHEMA sales.internal;     -- domain-private working tables
GRANT USE SCHEMA, SELECT ON SCHEMA sales.products TO `grp-analysts-all`;

Only products is visible to other domains; internal stays private.

#3. A self-serve platform (the central team's new job)

The central team becomes a platform team that provides:

• Templates: a repo template with dbt/DLT project layout, CI, tests and deployment.
• Compute policies and budgets per domain.
• Shared ingestion connectors and observability.
• Onboarding docs — a new domain should publish a first product in days, not months.

#4. Federated governance

Some rules are global and automated:

• Naming conventions and required tags (owner, classification) enforced in CI and with catalog tags.
• PII classification and masking policies applied centrally (column masks/row filters).
• Quality checks required before a table can be in a products schema.

ALTER TABLE sales.products.orders SET TAGS ('owner' = 'sales-data', 'classification' = 'internal', 'sla_freshness' = 'daily-06:00');

#5. Start small

1. Pick one domain with a motivated team and a clear consumer.
2. Publish two or three products with SLOs and documentation.
3. Move the matching central pipelines to that team, with the central team pairing.
4. Measure: lead time for new data requests, incidents, consumer satisfaction.
5. Repeat with the next domain.

Tip

Don't reorganise everything at once. A domain without data engineering skills needs a platform engineer embedded for a few months, or it will fail and you'll be back to the central model.

#Checklist

• Data product definition agreed
• Catalog/workspace boundaries per domain with public vs. private areas
• Platform templates, CI and compute policies ready
• Global governance automated (tags, PII masking, quality gates)
• First domain live; metrics tracked