Transformation: one shared Databricks workspace to dev/test/prod workspaces with catalog bindings

#Target layout

One metastore (per region)
 ├─ ws-data-dev   → bound to catalogs: dev        (read-only: prod_sample)
 ├─ ws-data-test  → bound to catalogs: test
 └─ ws-data-prod  → bound to catalogs: prod

Catalogs per environment (dev, test, prod) with the same schema names, so code changes only the catalog variable.

#1. Create workspaces

Deploy with Terraform/Bicep: VNet-injected, same network pattern, each with its own managed resource group. Assign all to the same metastore.

#2. Catalogs and storage

Give each environment catalog its own managed storage location (separate containers or accounts) so prod data isn't physically shared with dev:

CREATE CATALOG prod MANAGED LOCATION 'abfss://prod@stlakeprod.dfs.core.windows.net/';
CREATE CATALOG dev  MANAGED LOCATION 'abfss://dev@stlakedev.dfs.core.windows.net/';

#3. Bind catalogs to workspaces

Set the catalog's isolation mode to isolated and bind it, so prod is reachable only from the prod workspace even if someone has grants:

databricks catalogs update prod --isolation-mode ISOLATED
databricks workspace-bindings update-bindings catalog prod \
  --json '{"add":[{"workspace_id": 3333333333333333, "binding_type": "BINDING_TYPE_READ_WRITE"}]}'

A read-only binding of a sample/masked catalog to dev is a common way to give developers realistic data.

#4. Identities

• People: groups from Entra ID via SCIM at account level; assign to workspaces by group. Few people in prod workspace, mostly view rights.
• Jobs: a service principal per environment (sp-data-prod) owns and runs production jobs.

#5. Move existing assets

1. Put jobs and notebooks into bundles (see the asset bundles migration post).
2. Repoint table names from hive_metastore.x.y or hard-coded catalogs to ${var.catalog}.x.y.
3. Move production tables: if already in UC, tables can't be moved between catalogs in place — use deep clone or CTAS into the new catalog, or keep the current catalog as prod and create the new ones for dev/test.
4. Deploy to prod workspace with the service principal; disable the job in the old workspace.

#6. Retire the shared workspace

Once only dev activity remains, either make it the dev workspace or delete it after exporting what people still need.

Warning

Cluster-level credentials (storage keys in Spark config) bypass catalog bindings. Remove them as part of the split, or isolation is only on paper.

#Checklist

• Three workspaces on one metastore
• Environment catalogs with separate storage
• Prod catalog isolated and bound to prod only
• Jobs in bundles, run by environment service principals
• Old workspace emptied and retired