Moving Unity Catalog tables and volumes to a new ADLS storage account on Azure Databricks

#When you need this

A storage account must be replaced. Common reasons: it was created without hierarchical namespace, it's in the wrong region or subscription, it needs a different redundancy or encryption setup, or a catalog's managed storage sits on an account that is being retired. Unity Catalog has no "move" command for data. You build the new storage, copy the data in a controlled way, repoint the metadata, and re-grant.

This runbook assumes source account oldlake and target account newlake, both ADLS Gen2.

#Phase 0: inventory

Find everything that resolves to the old account:

-- External locations on the old account
SELECT external_location_name, url, storage_credential_name
FROM   system.information_schema.external_locations
WHERE  url LIKE 'abfss://%@oldlake.dfs.core.windows.net%';

-- Tables stored there (managed and external)
SELECT table_catalog, table_schema, table_name, table_type, storage_path
FROM   system.information_schema.tables
WHERE  storage_path LIKE 'abfss://%@oldlake.dfs.core.windows.net%';

-- Volumes
SELECT volume_catalog, volume_schema, volume_name, volume_type, storage_location
FROM   system.information_schema.volumes
WHERE  storage_location LIKE 'abfss://%@oldlake.dfs.core.windows.net%';

Also check:

• DESCRIBE CATALOG EXTENDED <catalog> and DESCRIBE SCHEMA EXTENDED <schema> for managed locations on the old account.
• Streaming checkpoints, Auto Loader schema locations and job code that use abfss:// paths to the old account directly. Search your repos for oldlake.dfs.core.windows.net.
• External readers such as Power BI, Synapse or Fabric shortcuts that point at the old paths.

Export the grants on every object you will recreate, before you change anything:

SHOW GRANTS ON TABLE prod.sales.orders;
-- or, in bulk:
SELECT * FROM system.information_schema.table_privileges
WHERE  table_catalog = 'prod';

#Phase 1: build the target

az storage account create -n newlake -g rg-data -l westeurope \
  --sku Standard_ZRS --kind StorageV2 --hns true \
  --min-tls-version TLS1_2 --allow-blob-public-access false
az storage container create --account-name newlake -n lake --auth-mode login

# Grant the Databricks access connector's managed identity data access
az role assignment create \
  --assignee-object-id <access-connector-principal-id> --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" \
  --scope $(az storage account show -n newlake -g rg-data --query id -o tsv)

Copy the network setup from the old account: private endpoints, resource instance rules for the access connector, and serverless NCC private endpoints if serverless reads this data.

CREATE EXTERNAL LOCATION newlake_lake
  URL 'abfss://lake@newlake.dfs.core.windows.net/'
  WITH (STORAGE CREDENTIAL ac_prod);

Open the new external location in Catalog Explorer and run Test connection before you copy anything.

#Phase 2: choose a method per object

Warning

Don't clone or copy shallow clones. They reference the source's data files by absolute path. Re-create them as shallow clones of the new tables.

#Managed tables: new schema on the new account

CREATE SCHEMA prod.sales_v2
  MANAGED LOCATION 'abfss://lake@newlake.dfs.core.windows.net/prod/sales';

CREATE TABLE prod.sales_v2.orders DEEP CLONE prod.sales.orders;

Re-run DEEP CLONE just before cutover. A repeat clone into the same target copies only new files, so the final sync is short.

#External Delta tables by file copy

Stop all writers first. Then:

azcopy login --tenant-id <tenant-id>
azcopy copy \
  'https://oldlake.dfs.core.windows.net/lake/ext/orders_raw/*' \
  'https://newlake.dfs.core.windows.net/lake/ext/orders_raw/' \
  --recursive

-- Compare before switching
DESCRIBE DETAIL delta.`abfss://lake@newlake.dfs.core.windows.net/ext/orders_raw`;

Check that numFiles and the latest version from DESCRIBE HISTORY match the source.

#Phase 3: cutover

1. Pause jobs, pipelines and streams that write to the affected objects.
2. Run the final incremental DEEP CLONE or AzCopy pass.
3. Validate row counts and the latest versions on the new copies.
4. Swap names. For managed tables in a new schema:

ALTER TABLE prod.sales.orders RENAME TO prod.sales.orders_old;
-- Point consumers at the new table, either by changing code to prod.sales_v2,
-- or by a view that keeps the old name:
CREATE VIEW prod.sales.orders AS SELECT * FROM prod.sales_v2.orders;

For external tables, dropping the table removes only metadata. Recreate it on the new path:

DROP TABLE prod.ext.orders_raw;
CREATE TABLE prod.ext.orders_raw
  LOCATION 'abfss://lake@newlake.dfs.core.windows.net/ext/orders_raw';

5. Re-apply the grants you exported in phase 0. They don't carry over to new or recreated objects. Reapply row filters, column masks, tags and comments too.
6. Restart the writers.

Caution

ALTER EXTERNAL LOCATION ... SET URL '<new>' FORCE changes only the metadata URL, even when tables depend on the location. It moves no data. Use it only when the data already sits at the same relative paths on the new account and you have tested how dependent objects resolve on a non-production copy. Creating a new external location is the safer default.

#Streaming checkpoints

A clone is a new Delta table with a new table ID. Streams that read the old table can't resume their checkpoint against the clone. For each affected stream:

(spark.readStream
   .option("startingVersion", "<first version not yet processed, or 'latest'>")
   .table("prod.sales_v2.orders")
   .writeStream
   .option("checkpointLocation", "/Volumes/prod/ops/checkpoints/orders_v2")  # new checkpoint
   .toTable("prod.gold.orders_enriched"))

Pick startingVersion so the stream neither skips nor reprocesses data. Make the sink idempotent (MERGE on keys) for the overlap. Auto Loader streams that read files from the old account need a new checkpoint and source path, and possibly modifiedAfter to avoid re-ingesting everything.

#Phase 4: decommission the old account

• Keep *_old tables and the old account read-only for an agreed period. Remove the access connector's write role first.
• Watch storage logs or the Unity Catalog audit logs for anything still reading the old paths.
• Then drop the old objects and external locations, and delete the account.

DROP TABLE prod.sales.orders_old;          -- managed: data is deleted after the retention period
DROP EXTERNAL LOCATION oldlake_lake;

#Rollback

Until phase 4, rollback means repointing consumers to the *_old tables, or recreating external tables on the old path, and restarting writers on their original checkpoints. Keep those checkpoints until the new streams have run cleanly for several cycles.

#Prevent it next time

• Put managed storage at the catalog or schema level on accounts you intend to keep for years, with the right region, redundancy and HNS from the start.
• Keep raw abfss:// paths out of job code. Use table names and /Volumes/... paths, so a storage move only touches Unity Catalog.
• Manage storage credentials, external locations and grants in Terraform, so re-granting at cutover is a re-apply rather than a manual rebuild.