Home / Data Platform / Governance

Data PlatformHow-toIntermediate

Advisory: environment strategy for data platforms - dev, test and prod without copying production data everywhere

Data platforms need separate environments like any software, but data makes it harder. Choose how environments are isolated, decide what data each one gets (synthetic, masked, sampled or read-only production), keep code promotion automated and identical, and stop sandboxes from becoming unmanaged copies of production.

Deepak Bhardwaj25 Oct 20262 min read

Applies toDatabricks workspaces/catalogs, Snowflake accounts/databases, Fabric workspaces, ADF factories, dbt targets, cloud subscriptions/accounts

Isolation options

Level	Example	Strength	Trade-off
Separate cloud subscriptions/accounts	prod and non-prod accounts	Strongest blast-radius and access separation	More setup, cross-account networking
Separate workspaces/accounts per env	Databricks workspace per env; Snowflake account per env	Clear separation, independent settings	Shared metadata needs care
Separate catalogs/databases in one platform	`dev_`, `test_`, `prod_` catalogs	Simple, cheap	Shared compute and admin; mistakes can cross

Common choice: prod isolated at subscription/account or workspace level; dev and test can share.

What data does each environment get?

Option	Use	Watch
Synthetic data	Unit tests, early dev	May not show real edge cases or volumes
Masked/tokenised copy	Integration and performance tests	Masking must be irreversible and maintained
Sample of production	Faster dev on realistic data	Still personal data unless masked
Read-only access to prod data from dev	Analytics development	Tight access control; no writes to prod; audit
Zero-copy clones (Snowflake, Delta shallow clone)	Fast test environments	Inherit data sensitivity; clean up

Decide per data class: personal or regulated data should not reach dev without masking.

Promotion

Practice	Why
All code in Git, deployed by CI/CD	Same artifact runs in every env
Parameterise env-specific values (catalog names, storage paths, secrets)	No code edits between envs
Service identities deploy and run prod jobs	No personal accounts in prod
Infrastructure as code for platform objects	Environments stay consistent
Approval gate before prod	Change control

# Databricks Asset Bundle targets (excerpt)
targets:
  dev:
    mode: development
    variables:
      catalog: dev
  prod:
    mode: production
    variables:
      catalog: prod
    run_as:
      service_principal_name: "sp-data-prod"

Sandboxes

Time-limited and owned; auto-clean after inactivity.
No production writes from sandboxes.
Cost tags per owner so spend is visible.
Data copies follow the same retention rules (see the retention guide).

Warning
"Temporary" full copies of production in dev are one of the most common data exposure findings. Prefer masked clones or controlled read access, and expire copies automatically.

Checklist

Isolation level chosen per environment
Data provisioning rules per data class
Masking in place for non-prod
CI/CD with parameterised configs and service identities
Infrastructure as code for platform objects
Sandbox ownership, expiry and cost tags