Map each cluster type
| HDInsight cluster | Target |
|---|---|
| Spark | Azure Databricks jobs/serverless |
| Hive / Interactive Query (LLAP) | Databricks SQL warehouses |
| Kafka | Azure Event Hubs (Kafka endpoint) or Confluent Cloud |
| HBase | Azure Cosmos DB (NoSQL / Cassandra API), or Delta tables for analytical access |
| Oozie workflows | Databricks Workflows or ADF |
| ML Services (R Server) | Databricks ML / Azure ML |
Check HDInsight version support and retirement dates for your clusters — they drive the timeline.
1. Data stays where it is
HDInsight clusters typically use ADLS Gen2 as primary storage. Databricks reads the same paths, so there's no bulk copy for most data. Register it in Unity Catalog:
CREATE EXTERNAL LOCATION hdi_data URL 'abfss://hdi@sthdinsight.dfs.core.windows.net/hive/warehouse/'
WITH (STORAGE CREDENTIAL sc_hdi);
2. Hive metastore → Unity Catalog
HDInsight uses an external Hive metastore (Azure SQL DB) or an internal one. Export table DDL:
-- On HDInsight (beeline)
SHOW CREATE TABLE sales.orders;
Recreate as UC external tables over the same locations. ORC tables: read as external tables or convert to Delta with CTAS (recommended for performance and updates):
CREATE TABLE main.sales.orders CLUSTER BY (order_date)
AS SELECT * FROM orc.`abfss://hdi@sthdinsight.dfs.core.windows.net/hive/warehouse/sales.db/orders`;
Hive ACID (transactional) tables need a compaction/export on HDInsight first, because their delta directories aren't plain readable files.
3. Spark jobs
- Update Spark versions (HDInsight Spark 3.1/3.3 → current DBR): check deprecated APIs and config changes.
- Replace
spark-submitwith job tasks (spark_python_task / JAR tasks). - Replace YARN-specific settings and HDFS paths.
4. Hive/LLAP queries
Most HiveQL runs in Databricks SQL. Watch out for Hive UDFs (Java), INSERT OVERWRITE DIRECTORY, and type coercion differences. Point BI tools (Power BI via the Databricks connector) at a SQL warehouse.
5. Kafka
Move producers/consumers to Event Hubs' Kafka endpoint (change bootstrap servers and SASL settings), or use MirrorMaker 2 to replicate during the switch. Check feature compatibility (transactions, compaction) first.
6. Decommission
Run workloads in parallel, reconcile, switch consumers, then delete clusters — and the metastore database and ESP/Entra DS resources if only HDInsight used them.
Tip
Delete clusters, not storage. HDInsight clusters can be deleted without touching ADLS data; keep the data until Databricks owns it fully.
Checklist
- Each cluster mapped to a target service
- Data registered in UC (or converted to Delta)
- Spark jobs and Hive queries converted and tested
- Kafka clients moved
- Clusters and metastore decommissioned