Home / Databricks on Azure

Databricks on AzureRunbookAdvanced

Transformation: upgrade from the Hive metastore to Unity Catalog with UCX

Move tables, permissions, jobs and code from the workspace Hive metastore to Unity Catalog using Databricks Labs UCX. Assessment, group migration, storage credentials and external locations, table upgrade, code changes and cutover.

Deepak Bhardwaj3 min read

Applies toAzure Databricks Premium workspaces, Unity Catalog, Databricks Labs UCX, Databricks CLI

What has to move

Hive-metastore world Unity Catalog world
Tables in hive_metastore.<db> <catalog>.<schema>.<table>
Table ACLs / credential passthrough UC grants, row filters, column masks
Mount points, SAS/keys in Spark config Storage credentials (access connector) + external locations, volumes
Workspace-local groups Account-level groups
Clusters without UC access mode Standard (shared) or dedicated (single user) access mode

1. Install UCX and assess

databricks labs install ucx
databricks labs ucx ensure-assessment-run

The assessment dashboard lists tables (managed/external, formats, locations), mounts, jobs and clusters that aren't UC-compatible, init scripts and code using deprecated patterns (DBFS paths, RDD APIs on shared clusters, sc._jsc hacks).

2. Groups

UC requires account-level groups. UCX migrates workspace-local group permissions to matching account groups:

databricks labs ucx migrate-groups      # check current command names in the UCX README for your version

Sync groups from Entra ID (SCIM or automatic identity management) first.

3. Storage access

Create an access connector (managed identity) with Storage Blob Data Contributor on the lake accounts, then storage credentials and external locations:

databricks labs ucx create-uber-principal          # optional helper
databricks labs ucx migrate-credentials
databricks labs ucx migrate-locations

Or manually:

CREATE EXTERNAL LOCATION raw_lake URL 'abfss://raw@stlake.dfs.core.windows.net/'
WITH (STORAGE CREDENTIAL sc_lake_connector);
GRANT READ FILES ON EXTERNAL LOCATION raw_lake TO `grp-data-engineers`;

4. Create catalogs and map tables

databricks labs ucx create-table-mapping         # produces a mapping file: workspace db → catalog.schema
databricks labs ucx create-catalogs-schemas
databricks labs ucx migrate-tables
Source table Upgrade method
External Delta/Parquet on ADLS SYNC — registers the same files in UC (no copy)
Managed table on DBFS root Deep clone / CTAS into a UC managed table (data copy)
Non-Delta formats Register as external, or convert 
SYNC SCHEMA main.sales FROM hive_metastore.sales DRY RUN;
SYNC SCHEMA main.sales FROM hive_metastore.sales;

5. Code and jobs

  • Replace hive_metastore.db.table / unqualified names with three-level names (or set the default catalog per workspace/job).
  • Replace /mnt/... paths with volumes (/Volumes/catalog/schema/volume/...) or table names.
  • Switch job clusters to a UC access mode; UCX's linter (databricks labs ucx lint-local-code) flags incompatible code.

6. Cut over

Run jobs in parallel against UC tables for a cycle where possible, then switch jobs, BI connections and users. Make the old HMS tables read-only (revoke MODIFY) and eventually drop them.

Warning

SYNC-ed external tables share files with the HMS table. Don't run writes from both sides at once, and don't drop the HMS table with a command that deletes data.

Checklist

  • UCX assessment reviewed; blockers listed
  • Account groups synced; permissions migrated
  • Access connector, storage credentials, external locations
  • Tables upgraded (SYNC/clone) and validated
  • Jobs/code moved to three-level names and volumes; HMS frozen

Related fixes