Sev1: a Terraform apply revoked Unity Catalog grants — users and jobs lose access to production data

#Symptoms

• Analysts, BI tools and jobs fail with PERMISSION_DENIED: User does not have USE CATALOG on Catalog 'prod' or SELECT on Table.
• It started right after a Terraform apply on the Databricks workspace or Unity Catalog configuration.

#1. Why Terraform removed grants

The Databricks provider has two styles:

If someone created databricks_grants for a catalog with only the new team's permissions, every other grant on that catalog (made manually or by another stack) was revoked.

#2. Find what was revoked

Query the audit system table:

SELECT event_time, user_identity.email AS who, action_name,
       request_params
FROM system.access.audit
WHERE service_name = 'unityCatalog'
  AND action_name = 'updatePermissions'
  AND event_time >= current_timestamp() - INTERVAL 1 DAY
ORDER BY event_time DESC;

request_params contains the securable and the changes (added/removed privileges per principal). That is your restore list. Also check the Terraform plan output in the pipeline logs: it lists the grants that were removed.

Current state:

SHOW GRANTS ON CATALOG prod;
SHOW GRANTS ON SCHEMA prod.sales;

#3. Restore access now

Re-grant the removed privileges with SQL (as the owner or a metastore admin):

GRANT USE CATALOG ON CATALOG prod TO `analysts`;
GRANT USE SCHEMA, SELECT ON SCHEMA prod.sales TO `analysts`;
GRANT USE CATALOG ON CATALOG prod TO `sp-etl-prod`;
GRANT USE SCHEMA, SELECT, MODIFY ON SCHEMA prod.sales TO `sp-etl-prod`;

Service principals are referenced by their application ID in grants.

Warning

If the next pipeline run applies the same Terraform again, it will revoke your manual grants a second time. Pause the pipeline or fix the code (step 4) before or immediately after restoring.

#4. Fix the Terraform

Option A: include all grants in the authoritative resource, so Terraform owns the full list:

resource "databricks_grants" "prod_catalog" {
  catalog = "prod"
  grant {
    principal  = "analysts"
    privileges = ["USE_CATALOG"]
  }
  grant {
    principal  = "sp-etl-prod-app-id"
    privileges = ["USE_CATALOG"]
  }
  grant {
    principal  = "data-team-new"
    privileges = ["USE_CATALOG"]
  }
}

Option B: switch to databricks_grant (singular) per principal, so stacks don't fight:

resource "databricks_grant" "new_team_catalog" {
  catalog    = "prod"
  principal  = "data-team-new"
  privileges = ["USE_CATALOG"]
}

Never mix databricks_grants and other grant management on the same securable.

#5. Verify

-- as an affected user or with a test query from the job's service principal
SELECT COUNT(*) FROM prod.sales.orders;

Rerun failed jobs and refresh BI datasets.

#Prevent it next time

• Grants for a securable are managed in exactly one place.
• Plans that remove grants on production catalogs require explicit approval (policy check in CI, for example with OPA/Conftest on the plan JSON).
• Grant to groups, not individuals, so lists stay short and reviewable.
• Alert on updatePermissions events that remove privileges in production catalogs.