A Delta lakehouse blueprint for Azure Databricks: catalogs, medallion layers, loading patterns and data products

Most lakehouse problems aren't Spark problems. They're design problems: no agreed layout, every team loading data differently, and nobody sure which table is "the real one". This blueprint gives the decisions to make once, so every pipeline follows the same pattern.

#1. The big picture

Sources ──> Landing (files) ──> Bronze ──> Silver ──> Gold ──> Consumers
            ADF / Lakeflow      raw        clean      business   Power BI, SQL,
            Connect / Kafka     history    conformed  products   ML, Delta Sharing

#2. Unity Catalog layout

Choose one convention and enforce it. Two that work well:

A. Catalog per environment + domain, schema per layer (most common):

dev_sales.bronze      test_sales.bronze      prod_sales.bronze
dev_sales.silver      test_sales.silver      prod_sales.silver
dev_sales.gold        test_sales.gold        prod_sales.gold

B. Catalog per environment + layer, schema per domain: prod_silver.sales, prod_silver.finance.

Pattern A keeps domain ownership clear: one owner group per catalog. Pattern B suits a central platform team.

Whichever you choose:

• Workspace-catalog binding: prod_* catalogs are only bound to prod workspaces.
• Managed location per catalog, each in its own ADLS container, for isolation and cost tracking.
• Owner = a group, for example sales-data-owners. Pipelines write as a service principal.
• Shared reference data (calendar, currency, org hierarchy) goes in a prod_shared catalog.

#Naming

#3. Table standards

Apply them with a template or in a shared library (see the Python wheel post):

CREATE TABLE prod_sales.silver.orders (
  order_id      STRING NOT NULL,
  customer_id   STRING,
  order_ts      TIMESTAMP,
  amount        DECIMAL(18,2),
  status        STRING,
  _ingest_ts    TIMESTAMP,
  _batch_id     STRING,
  CONSTRAINT orders_pk PRIMARY KEY (order_id)          -- informational, helps BI and the optimizer
)
CLUSTER BY (order_ts, customer_id)                      -- liquid clustering on common filters
COMMENT 'One row per sales order; latest state. Source: SAP VBAK/VBAP.'
TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true');

ALTER TABLE prod_sales.silver.orders ADD CONSTRAINT valid_amount CHECK (amount >= 0);

• Managed tables by default. Enable predictive optimization so OPTIMIZE and VACUUM run automatically.
• Liquid clustering instead of partitioning for most tables. Partition only very large tables with a natural, low-cardinality key such as date, and only if you need it.
• Comments on tables and key columns. They feed search, AI/BI Genie and documentation.
• Change Data Feed on silver tables that feed incremental gold.
• Tags for classification (pii, confidential) to drive masking and policies.

#4. Standard loading patterns

Give every pipeline one of these five patterns. Don't invent new ones.

-- partition replace: recompute the last 3 days idempotently
INSERT INTO prod_sales.gold.agg_sales_daily
REPLACE WHERE order_date >= current_date() - 3
SELECT to_date(order_ts) AS order_date, region, count(*) AS orders, sum(amount) AS revenue
FROM prod_sales.silver.orders o JOIN prod_sales.silver.customers c USING (customer_id)
WHERE to_date(order_ts) >= current_date() - 3
GROUP BY ALL;

Every load must be idempotent. Rerunning the same batch gives the same result. Then retries and backfills are safe.

#5. Data quality

In Lakeflow declarative pipelines, use expectations:

CONSTRAINT valid_key    EXPECT (order_id IS NOT NULL)  ON VIOLATION DROP ROW,
CONSTRAINT valid_amount EXPECT (amount >= 0)           ON VIOLATION FAIL UPDATE

For continuous checks such as freshness and anomalies, add data quality monitoring (Lakehouse Monitoring) on key gold tables.

#6. Compute per workload

#7. Gold as data products

A gold table is a product with an owner and a contract, not just a table.

• Owner group and support channel in the table comment or tags.
• Documented grain, keys and refresh schedule (SLA).
• Stable schema. Breaking changes go to a new table or version (fact_sales_v2) with a deprecation period.
• Access granted to consumer groups; row filters and masks where needed.
• Shared outside the platform with Delta Sharing, not file exports.
• Certified with tags (for example certified=true), so users know which tables to trust.

#8. Operations

• Lineage comes automatically in Unity Catalog. Use it for impact analysis before changing silver tables.
• System tables for cost per catalog and job, and for failed runs (see the monitoring post).
• Time travel retention of at least 7 days on silver and gold; deep clone backups for critical tables.
• CI/CD with Asset Bundles: dev, then test, then prod, with the same code and different catalog names passed as variables.

# bundle variable makes the same pipeline target dev_sales / prod_sales
variables:
  catalog:
    default: dev_sales
targets:
  prod:
    variables:
      catalog: prod_sales

#Anti-patterns

Tip

Write these decisions down as a one-page lakehouse standard in your repository: layout, naming, the five loading patterns and the data-product checklist. Review new pipelines against it in pull requests. Consistency matters more than any single choice.