Home / Data Platform / Databricks & Spark

Data PlatformHow-toIntermediate

Databricks platform foundation: workspaces, Unity Catalog metastore and catalogs per environment

Set up one metastore per region, workspaces per environment, catalogs bound to workspaces, storage credentials and external locations, groups from your IdP and cluster policies.

Deepak Bhardwaj2 min read

Applies toDatabricks on Azure, AWS and GCP with Unity Catalog

1. Topology

Account (Databricks account console)
└── Metastore: one per region (e.g. centralindia)
    ├── Workspace: dbw-dev   → catalog dev     (bound)
    ├── Workspace: dbw-test  → catalog test    (bound)
    └── Workspace: dbw-prod  → catalog prod    (bound)
  • One metastore per region; all workspaces in that region attach to it.
  • Separate workspaces per environment (and sometimes per business unit).
  • Workspace-catalog binding: the prod catalog is only accessible from the prod workspace, even for admins in dev.

2. Identity

  • Users and groups provisioned at the account level via SCIM from Entra ID/Okta.
  • Assign groups to workspaces; grant data permissions to groups, never individuals.
  • Service principals for jobs and CI/CD (OAuth M2M or workload identity federation).

3. Storage credentials and external locations

-- created by an admin; the credential wraps a managed identity / IAM role
CREATE EXTERNAL LOCATION IF NOT EXISTS prod_lake
  URL 'abfss://lake@stlakeprodcin.dfs.core.windows.net/'
  WITH (STORAGE CREDENTIAL sc_lake_prod);

CREATE CATALOG IF NOT EXISTS prod MANAGED LOCATION 'abfss://lake@stlakeprodcin.dfs.core.windows.net/uc/prod';
GRANT USE CATALOG ON CATALOG prod TO `data-analysts`;

Users never get direct storage access; access goes through Unity Catalog grants.

4. Cluster and compute policies

  • Cluster policies restrict node types, max workers, auto-termination and runtime versions.
  • Use job clusters for scheduled jobs; serverless or SQL warehouses for BI.
  • Tag compute with cost centre for chargeback (system tables system.billing.usage).
{
  "autotermination_minutes": { "type": "range", "maxValue": 60, "defaultValue": 30 },
  "spark_version": { "type": "regex", "pattern": "1[4-9]\\.[0-9]+\\.x-scala.*" },
  "custom_tags.CostCenter": { "type": "fixed", "value": "CC-4512" }
}

5. Networking

  • Workspace with VNet/VPC injection and no public IPs (secure cluster connectivity).
  • Private Link for front-end (UI/API) and back-end where required.
  • Storage accessed via private endpoints.

6. Observability

Enable system tables (audit, billing, lineage, query history) and build a small operations dashboard: job failures, DBU spend by team, long-running clusters.

DR foundations

Keep everything as code (Terraform Databricks provider, Databricks Asset Bundles for jobs/notebooks). Workspace objects aren't automatically replicated to another region; code is how you rebuild (see the Databricks DR post).

Related fixes