Home / Data Platform / Databricks & Spark

Data PlatformHow-toAdvanced

Moving Delta data between Databricks tenants: storage copy, Delta Sharing or deep clone

Choose and run the right method to copy lakehouse data into the target tenant. Covers re-pointing to existing storage, copying Delta folders with azcopy or S3 replication and re-registering tables, Databricks-to-Databricks Delta Sharing with CTAS, incremental sync, and validation.

Deepak Bhardwaj3 min read

Applies toAzure Databricks (ADLS Gen2) and Databricks on AWS (S3) with Unity Catalog; source and target in different tenants/accounts

1. Options

Option Speed History kept When
A. Keep the storage, re-point (move the storage account/bucket to the target, or grant the target access) No copy Yes Storage can move with the company, or cross-tenant access is allowed temporarily
B. Storage-level copy (azcopy / ADF / S3 Batch Replication) + register tables Fast, parallel Yes (Delta log copied) Large volumes, clean cut
C. Delta Sharing (D2D) + CTAS / MERGE in target Moderate No (new table history) No storage access across tenants; selective data; incremental sync

2. Option A: keep the storage

  • External tables in the source can be registered as external tables in the target UC once the target's access connector/IAM role can read the storage (cross-tenant managed identity access isn't possible directly; you'd typically move the storage account's subscription to the target tenant, then grant the new access connector).
  • Managed tables in the source live in the source metastore's managed storage — use B or C for those.

3. Option B: copy files, then register

Copy the whole table folder (data files and _delta_log):

azcopy copy "https://stsrclake.dfs.core.windows.net/silver/sales/orders/?<src-sas>" \
            "https://sttgtlake.dfs.core.windows.net/silver/sales/orders/?<tgt-sas>" --recursive

For managed tables, find the storage path with DESCRIBE DETAIL and copy that folder. Then register in the target:

CREATE TABLE prod_sales.orders.orders USING DELTA
LOCATION 'abfss://silver@sttgtlake.dfs.core.windows.net/sales/orders';

This creates an external table with full history. If you want UC managed tables, CREATE TABLE ... DEEP CLONE from the external table into a managed catalog, then drop the external copy (or use the external-to-managed conversion command where available in your runtime).

Copy while writes are paused, or copy and then re-copy changed files (azcopy sync) at cutover. Never copy a table while it's being vacuumed.

4. Option C: Delta Sharing between accounts

  1. In the source: create a recipient for the target metastore's sharing identifier; create a share with the tables (enable history sharing / CDF if you want incremental reads).
CREATE SHARE migr_sales;
ALTER SHARE migr_sales ADD TABLE prod.sales.orders WITH HISTORY;
CREATE RECIPIENT newco_metastore USING ID 'azure:centralindia:<target-metastore-uuid>';
GRANT SELECT ON SHARE migr_sales TO RECIPIENT newco_metastore;
  1. In the target: create a catalog from the share, then materialise:
CREATE CATALOG src_share USING SHARE oldco_provider.migr_sales;
CREATE TABLE prod_sales.orders.orders AS SELECT * FROM src_share.sales.orders;
  1. Incremental sync until cutover: MERGE using change data feed (table_changes) from the shared table, or reload partitions.

5. Validate

SELECT COUNT(*), SUM(hash(*)) FROM prod.sales.orders;   -- run in source and target and compare

Plus schema comparison (DESCRIBE TABLE), partition counts, and spot checks on recent data. Check table properties (CDF, retention, clustering) were reapplied.

6. Plan the volume

Measure throughput with one large table first. Copy cold, large tables early; leave small, hot tables for the cutover window.

Warning

Copying _delta_log and data files separately or out of order can produce a broken table (log entries pointing to missing files). Copy the full folder in one operation, and verify with DESCRIBE HISTORY and a full read in the target.