Component mapping
| Hadoop | Lakehouse |
|---|---|
| HDFS | ADLS Gen2 / S3 with Delta tables |
| Hive metastore, Hive tables | Unity Catalog, Delta (managed or external) |
| Hive/Impala SQL | Databricks SQL warehouses |
| Spark on YARN | Databricks jobs (job clusters or serverless) |
| MapReduce / Pig | Rewrite in Spark SQL / PySpark |
| Oozie | Databricks Workflows, or Airflow/ADF |
| Ranger / Sentry | Unity Catalog grants, row filters, column masks |
| Kafka + Spark Streaming | Structured Streaming / Lakeflow pipelines |
| HBase | Delta tables for analytics; Cosmos DB/DynamoDB for low-latency lookups |
| Sqoop | Lakeflow Connect, ADF, or JDBC reads in Spark |
1. Inventory and prioritise
- Tables: size, partitions, file formats, update patterns, consumers.
- Jobs: Oozie workflows, crontabs on edge nodes, their SLAs and owners.
- Prioritise by value and simplicity: start with a business domain whose consumers can test quickly.
2. Land the data
Copy HDFS data to cloud storage (DistCp over ExpressRoute/Direct Connect, Azure Data Box / AWS Snowball for very large volumes), then convert:
-- Parquet already in cloud storage: convert in place
CONVERT TO DELTA parquet.`abfss://raw@stlake.dfs.core.windows.net/warehouse/sales` PARTITIONED BY (sale_date DATE);
-- ORC/Avro/text: rewrite
CREATE TABLE main.sales.transactions
CLUSTER BY (sale_date, store_id)
AS SELECT * FROM orc.`abfss://raw@stlake.dfs.core.windows.net/warehouse/transactions_orc`;
Compact small files during the rewrite; Hadoop estates usually have millions of them.
3. Convert code
- Hive/Impala SQL: mostly runs on Spark SQL. Watch for UDFs (Java JARs — port or re-register),
INSERT OVERWRITEpartition semantics, implicit type casts and Impala-specific functions. - Spark jobs: update Spark version APIs (Spark 1.x/2.x → 3.x), replace
hdfs://paths with Unity Catalog volumes or table names, drop YARN-specific settings. - MapReduce/Pig: rewrite; usually much shorter in Spark SQL.
Automated SQL conversion tools (for example Databricks' acquired converters or partner tools) speed up large estates; review the output.
4. Orchestration and security
- Rebuild Oozie workflows as Databricks Workflows (tasks, dependencies, retries, schedules) or in your existing Airflow.
- Translate Ranger policies to Unity Catalog grants on catalogs/schemas, plus row filters and column masks for fine-grained rules.
5. Parallel run and switch
Run old and new for one or two business cycles. Compare outputs table by table (row counts, sums of key measures, sample record hashes). Switch consumers (BI tools, extracts) domain by domain, then freeze the Hadoop tables as read-only and decommission nodes in stages.
Tip
Keep one person accountable for each domain's reconciliation sign-off. Migrations stall when "the numbers are slightly different" has no owner.
Checklist
- Component map and domain-by-domain plan
- Data in cloud storage converted to Delta and compacted
- SQL and Spark code converted and tested
- Workflows and Unity Catalog permissions rebuilt
- Parallel run reconciled; Hadoop decommissioned