Home / Databricks on Azure

Databricks on AzureHow-toAdvanced

Advisory: data classification in Unity Catalog - governed tags, ABAC policies and keeping sensitive data protected at scale

Writing row filters and column masks table by table doesn't scale. Define a classification scheme as governed tags, tag columns automatically and by review, attach attribute-based access control policies to tags at catalog or schema level, and monitor for untagged sensitive data.

Deepak Bhardwaj2 min read

Applies toUnity Catalog governed tags, attribute-based access control (ABAC) policies, data classification, row filters and column masks on Azure Databricks; check feature availability in your region

Why tags plus policies

Table-by-table controls Tag-driven policies
Each new table needs a filter/mask attached New tables inherit protection once columns are tagged
Easy to miss a table Coverage measurable by tag queries
Logic copied everywhere One policy, many tables

Classification scheme

Keep it small and clear:

Tag key Values Meaning
classification public, internal, confidential, restricted Overall sensitivity
pii email, phone, national_id, name, address Type of personal data
region eu, in, us Residency-relevant data

Governed tags restrict who can apply them and which values are allowed, so the scheme stays consistent.

Tagging

ALTER TABLE prod.crm.customers ALTER COLUMN email SET TAGS ('pii' = 'email');
ALTER TABLE prod.crm.customers ALTER COLUMN national_id SET TAGS ('pii' = 'national_id', 'classification' = 'restricted');
  • Use automated data classification to suggest tags on new tables; a data steward confirms.
  • Make tagging part of the definition of done for new gold/silver tables.

Policies on tags

Example idea: mask any column tagged pii for users not in a privileged group, and filter rows by region based on group membership. ABAC policies attach at catalog, schema or table level and match columns by tag.

-- Masking function referenced by a policy
CREATE FUNCTION prod.governance.mask_pii(val STRING)
RETURNS STRING
RETURN CASE WHEN is_account_group_member('dbx-pii-readers') THEN val ELSE '***' END;

Create the policy that applies this function to columns with tag pii using the ABAC policy syntax or UI for your runtime; syntax is evolving, so follow current docs.

Monitor coverage

-- Columns with names that look sensitive but have no pii tag
SELECT c.table_catalog, c.table_schema, c.table_name, c.column_name
FROM system.information_schema.columns c
LEFT JOIN system.information_schema.column_tags t
  ON  c.table_catalog = t.catalog_name AND c.table_schema = t.schema_name
  AND c.table_name = t.table_name AND c.column_name = t.column_name AND t.tag_name = 'pii'
WHERE c.table_catalog = 'prod'
  AND regexp_like(lower(c.column_name), 'email|phone|ssn|aadhaar|pan|dob|address')
  AND t.tag_name IS NULL;

Operating model

  • Data stewards per domain own tagging.
  • Platform team owns policies and masking functions.
  • Quarterly review of exceptions and coverage.

Warning

Policies protect only tagged columns. Treat untagged sensitive columns as an incident to fix, and test policies with real user groups before relying on them.

Checklist

  • Small tag scheme defined as governed tags
  • Tagging automated with steward review
  • ABAC policies attached at catalog/schema level
  • Masking functions tested with real groups
  • Coverage monitored with scheduled queries
  • Ownership of tags and policies assigned