Databricks service deployment checklist: best practices for a secure, governed platform

Use this as a design review before the first workspace goes live. Each item should have an owner and a recorded decision.

#1. Account and workspace design

• One Databricks account per tenant/organisation; account admins are a group with PIM/JIT access.
• Workspace layout decided: per environment (dev / test / prod) at minimum; per business domain if isolation is needed.
• Region(s) chosen near data and users; DR region identified.
• Azure: Premium tier (needed for UC, RBAC, Private Link). AWS: Enterprise/Premium plan with the features you need.
• Naming and tagging standard (dbw-{domain}-{env}-{region}; tags env, owner, costCenter).
• Everything deployed with Terraform (cloud resources + Databricks objects).

#2. Networking

• Azure: VNet injection — two dedicated subnets (host and container), delegated to Microsoft.Databricks/workspaces, sized for max nodes (each node uses an IP in each subnet; /22 or larger for busy workspaces).
• AWS: customer-managed VPC with private subnets across 2+ AZs.
• Secure cluster connectivity (no public IPs on nodes) — default in new workspaces; confirm it's on.
• Private Link: back-end (compute plane ↔ control plane) and front-end (users/BI ↔ workspace) with private DNS; or front-end via a transit VNet.
• Public network access disabled, or IP access lists if a public front end is allowed.
• Egress through a firewall (Azure Firewall / NAT + filtering) with the required Databricks endpoints allowed; user-defined routes tested.
• Serverless egress controls: network connectivity configurations (NCC) and serverless network policies; private endpoints from serverless to your storage.

#3. Identity and access

• Entra ID / IdP integration: automatic identity management or SCIM provisioning; groups only.
• SSO enforced; no local passwords.
• Workspace entitlements per group; cluster creation only through policies.
• Jobs run as service principals; OAuth M2M or workload identity federation; PAT creation disabled or limited with max lifetime.
• Admin roles minimal (account admin, metastore admin, workspace admin) and reviewed quarterly.

#4. Unity Catalog and storage

• One metastore per region, owned by an admin group; workspaces assigned.
• Catalog strategy: <env>_<domain> catalogs; managed storage location per catalog (separate storage accounts/buckets or containers per domain/env).
• Storage credentials via access connector (Azure managed identity) / IAM role; no account keys in code.
• External locations only for paths that need them; least-privilege grants on them.
• Workspace-catalog bindings: prod catalogs only bound to prod workspaces.
• No DBFS mounts or hive_metastore for new work; legacy HMS disabled in new workspaces where possible.
• Grants to groups at catalog/schema level; row filters and column masks for sensitive data; tags for classification.
• Storage: hierarchical namespace (ADLS Gen2), private endpoints, soft delete/versioning, firewall to trusted networks only.

#5. Compute governance

• Cluster policies per persona (personal, shared, job, ML/GPU) with auto-termination, approved node types and runtimes, mandatory tags.
• Access modes: standard (shared) or dedicated (single user) — no "no isolation" clusters.
• Approved LTS runtime versions and an upgrade cadence.
• Pools for frequently started job clusters; spot with on-demand fallback for workers.
• SQL warehouses: serverless where allowed, auto-stop set, separate warehouses for BI / ad-hoc / ELT.
• Init scripts stored in UC volumes or workspace files (not DBFS), and allow-listed.

#6. Security and compliance

• Customer-managed keys where required: managed services, managed disks / workspace storage (Azure), workspace storage and EBS (AWS).
• Compliance security profile / enhanced security monitoring if regulated (HIPAA, PCI-DSS) — check availability for your cloud and region.
• Audit logs: system.access.audit enabled; diagnostic settings (Azure) or log delivery (AWS) to the SIEM.
• Secrets in Key Vault-backed scopes (Azure) or Databricks secret scopes fed from your vault; no secrets in notebooks.
• Data exfiltration controls: disable download of results/notebook export for sensitive workspaces if needed; restrict external Delta Sharing recipients.

#7. CI/CD and SDLC

• Git folders for all code; branch protection and pull requests.
• Databricks Asset Bundles for jobs, pipelines and dashboards; deploy to test then prod from CI with a service principal.
• Terraform state in a secured backend; plan/apply through pipelines with approval for prod.
• Unit tests (pytest) and data quality checks (expectations in DLT/Lakeflow, or Great Expectations) in the pipeline.

#8. Monitoring and operations

• System tables enabled (access, billing, compute, lakeflow, query); ops group has read access.
• Alerts: job failures, data freshness SLAs, warehouse queueing, security events.
• Ops dashboard and on-call runbooks (cluster start failures, job failures, permission denied).
• Support plan with Databricks and the cloud provider; escalation contacts documented.

#9. Cost

• Budgets and alerts per workspace/team; serverless budget policies.
• Tags enforced; monthly cost review with owners.
• Commit plan (DBCU pre-purchase / committed use) once usage is stable.

#10. Reliability and DR

• Job retries, timeouts and idempotent writes (see the job reliability post).
• Backup/recovery for critical tables (time travel retention, deep clone backups, storage soft delete).
• DR design: second-region workspace with IaC, data replication method, and a tested runbook (see the DR post).

Tip

Put this checklist in your repository as docs/platform-decisions.md with one line per item and the decision taken. When auditors or new team members ask "why is it built like this?", the answer is already written down.