Automating Azure Databricks with Terraform: workspace, Unity Catalog objects and grants as code

Clicking together one workspace is fine. By the third environment, catalogs drift, grants differ, and nobody knows why prod has an external location that dev lacks. Terraform keeps it consistent, if the code is structured well.

#Layer the configuration

Separate state per layer (and per environment). A mistake in the grants layer then cannot plan a workspace replacement.

#Providers

provider "azurerm" {
  features {}
}

provider "databricks" {
  alias      = "account"
  host       = "https://accounts.azuredatabricks.net"
  account_id = var.databricks_account_id
  # auth from environment: ARM_CLIENT_ID / ARM_TENANT_ID with OIDC, or azure-cli locally
}

provider "databricks" {
  alias = "workspace"
  host  = azurerm_databricks_workspace.this.workspace_url
}

Use Entra ID authentication (Azure CLI locally, workload identity federation in pipelines). Avoid personal access tokens in CI.

#Azure layer essentials

resource "azurerm_databricks_access_connector" "uc" {
  name                = "dbac-sales-prod"
  resource_group_name = azurerm_resource_group.this.name
  location            = var.location
  identity { type = "SystemAssigned" }
}

resource "azurerm_role_assignment" "uc_blob" {
  scope                = azurerm_storage_account.lake.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_databricks_access_connector.uc.identity[0].principal_id
}

#Data layer: credential, location, catalog

resource "databricks_storage_credential" "lake" {
  provider = databricks.workspace
  name     = "sc-sales-prod"
  azure_managed_identity {
    access_connector_id = azurerm_databricks_access_connector.uc.id
  }
  comment = "Managed by Terraform"
}

resource "databricks_external_location" "lake" {
  provider        = databricks.workspace
  name            = "el-sales-prod"
  url             = "abfss://lake@${azurerm_storage_account.lake.name}.dfs.core.windows.net/"
  credential_name = databricks_storage_credential.lake.name
  depends_on      = [azurerm_role_assignment.uc_blob]
}

resource "databricks_catalog" "sales" {
  provider       = databricks.workspace
  name           = "sales_prod"
  storage_root   = "${databricks_external_location.lake.url}managed/sales_prod"
  isolation_mode = "ISOLATED"
}

resource "databricks_schema" "silver" {
  provider     = databricks.workspace
  catalog_name = databricks_catalog.sales.name
  name         = "silver"
}

isolation_mode = "ISOLATED" plus a databricks_workspace_binding limits the catalog to the prod workspace.

#Grants: authoritative vs additive

resource "databricks_grants" "silver" {
  provider = databricks.workspace
  schema   = "${databricks_catalog.sales.name}.${databricks_schema.silver.name}"
  grant {
    principal  = "sales-engineers"
    privileges = ["USE_SCHEMA", "SELECT", "MODIFY", "CREATE_TABLE"]
  }
  grant {
    principal  = "sales-analysts"
    privileges = ["USE_SCHEMA", "SELECT"]
  }
}

Caution

Introducing databricks_grants on an existing catalog or schema removes every grant someone added by hand. Export current grants first (SHOW GRANTS ON SCHEMA ...), add them to code, and read the plan carefully. Use databricks_grant when several teams manage grants on the same object.

#Import existing objects

terraform import 'databricks_catalog.sales' sales_prod
terraform import 'databricks_schema.silver' sales_prod.silver
terraform import 'databricks_external_location.lake' el-sales-prod

Newer Terraform versions support import blocks, which are reviewable in a PR. The Databricks Terraform exporter (terraform-provider-databricks exporter) can generate code for an existing workspace as a starting point.

#Pipeline

1. terraform fmt -check, terraform validate, tflint.
2. terraform plan -out plan.tfplan per layer and environment; publish the plan.
3. Gate: any destroy or replace of catalogs, schemas, external locations, storage credentials, or the workspace requires manual approval.

terraform show -json plan.tfplan | jq -r '.resource_changes[] | select(.change.actions|index("delete")) | .address'

4. Apply with a service principal that is a metastore admin (data layer) or account admin (account layer), never a person.

#Protect what you cannot recreate

resource "databricks_catalog" "sales" {
  # ...
  force_destroy = false
  lifecycle { prevent_destroy = true }
}

Do the same for storage accounts holding managed tables and for the workspace itself.

#Common problems

#Prevent it next time

• One state per layer per environment.
• prevent_destroy on catalogs, credentials, locations and storage.
• Grants only through Terraform for production catalogs; read access for humans to propose changes via PR.