Databricks migration inventory: export everything you need to know with the CLI, APIs and system tables

#1. Set up access

databricks auth login --host https://adb-1111111111111111.11.azuredatabricks.net --profile src
databricks current-user me -p src

Use a workspace admin (and account admin for account-level objects).

#2. Workspace objects

databricks jobs list -p src --output json > inv/jobs.json
databricks clusters list -p src --output json > inv/clusters.json
databricks cluster-policies list -p src --output json > inv/policies.json
databricks instance-pools list -p src --output json > inv/pools.json
databricks warehouses list -p src --output json > inv/warehouses.json
databricks pipelines list-pipelines -p src --output json > inv/pipelines.json
databricks secrets list-scopes -p src --output json > inv/secret-scopes.json
databricks workspace list / -p src --output json > inv/workspace-root.json
databricks repos list -p src --output json > inv/repos.json

Count notebooks per folder (recursive listing via the Workspace API or databricks workspace export-dir dry runs) to size the code move.

#3. Unity Catalog and Hive metastore

-- UC tables, type and location
SELECT table_catalog, table_schema, table_name, table_type, data_source_format, storage_path
FROM system.information_schema.tables
WHERE table_catalog NOT IN ('system');

-- grants
SELECT * FROM system.information_schema.table_privileges;
SELECT * FROM system.information_schema.schema_privileges;

Hive metastore (legacy) — list databases and tables per database:

SHOW DATABASES IN hive_metastore;
SHOW TABLES IN hive_metastore.sales;
DESCRIBE TABLE EXTENDED hive_metastore.sales.orders;   -- location, format

Databricks Labs UCX produces a full HMS assessment (tables, mounts, incompatible code) if you haven't adopted UC yet.

#4. Size of data

DESCRIBE DETAIL prod.sales.orders;   -- sizeInBytes, numFiles

Script it over all tables and sum by catalog; this drives the data copy plan and egress cost.

#5. What is actually used

-- jobs run in the last 90 days
SELECT job_id, COUNT(*) AS runs, MAX(period_start_time) AS last_run
FROM system.lakeflow.job_run_timeline
WHERE period_start_time > current_date() - INTERVAL 90 DAYS
GROUP BY job_id;

-- tables read in the last 90 days (lineage)
SELECT source_table_full_name, COUNT(*) AS reads, MAX(event_time) AS last_read
FROM system.access.table_lineage
WHERE event_time > current_date() - INTERVAL 90 DAYS AND source_table_full_name IS NOT NULL
GROUP BY source_table_full_name;

Anything not used in 90 days → candidate to retire, not migrate.

#6. Hard-coded dependencies

Export the code and search it:

databricks workspace export-dir /Workspace ./src-export -p src
grep -rEn "dbfs:/mnt|abfss://|wasbs://|s3a?://|adb-[0-9]+|@contoso\.com|spark\.conf\.set\(.fs\.azure" ./src-export | tee inv/hardcoded.txt

Also list mounts: display(dbutils.fs.mounts()) in a notebook.

#7. Owners and identities

• Job owners and run_as identities (jobs.json), cluster creators, table owners (information_schema.tables.table_owner).
• Groups and members from the account console / SCIM API.

#8. The tracker

One row per asset: type, name, owner, last used, size, dependencies, target name, wave, status. This is the backbone of the whole migration.

Tip

Run the inventory twice: at the start and again a week before cutover. Teams keep creating jobs and tables during a migration, and the second run catches them.