The big change: storage and compute apart
On-premises Hadoop couples storage (HDFS) and compute on the same nodes, so clusters are sized for the larger of the two and run 24/7. On AWS, data lives in S3, metadata in the Glue Data Catalog, and compute (EMR clusters or EMR Serverless) runs only when jobs run.
1. Assess
- HDFS size and growth, small-file count, hot vs. cold data.
- Job inventory: Spark, Hive, Pig, MapReduce, Oozie workflows, HBase, Impala.
- Security: Kerberos, Ranger policies, encryption zones.
| On-prem component | AWS target |
|---|---|
| HDFS | S3 |
| Hive metastore | Glue Data Catalog (or external RDS metastore) |
| Spark / Hive jobs | EMR on EC2, EMR Serverless |
| Oozie | Step Functions, MWAA (Airflow) |
| Ranger | Lake Formation permissions |
| HBase | HBase on EMR with S3 storage, or DynamoDB |
| Impala | Athena or Trino on EMR |
2. Move the data
Over Direct Connect with enough bandwidth, DistCp from the source cluster:
hadoop distcp -Dfs.s3a.fast.upload=true -m 100 -update \
hdfs:///warehouse/sales s3a://corp-datalake-raw/warehouse/sales
Or use AWS DataSync with an HDFS location (agent near the cluster). For hundreds of TB with limited bandwidth, consider an offline transfer device.
Compact small files during the move (write larger Parquet files) — S3 and Spark both perform better with files in the hundreds of MB.
3. Move metadata
Export Hive DDL and recreate tables in Glue with S3 locations, or run the AWS-provided Hive-metastore-to-Glue migration scripts. For EMR:
[{
"Classification": "spark-hive-site",
"Properties": { "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory" }
}]
4. Adapt jobs
- Replace
hdfs://paths withs3://URIs. - Remove assumptions about rename being cheap (S3 rename is copy+delete); use the EMRFS S3-optimized committer for Spark output.
- Consider open table formats (Iceberg, Delta, Hudi) for updates and deletes instead of rewriting partitions.
5. Run compute the cloud way
aws emr-serverless start-job-run --application-id 00abc --execution-role-arn arn:aws:iam::111122223333:role/emr-job \
--job-driver '{"sparkSubmit":{"entryPoint":"s3://corp-code/jobs/daily_sales.py","sparkSubmitParameters":"--conf spark.executor.memory=8g"}}'
- EMR Serverless for scheduled batch jobs: no cluster to manage.
- Transient EMR on EC2 clusters (with Spot for task nodes) for jobs needing specific configuration.
- A small long-running cluster only for interactive workloads.
6. Validate and switch
Run jobs in parallel on both platforms for a cycle and compare row counts and aggregates per table. Switch downstream consumers (BI tools to Athena/Trino), then decommission the Hadoop cluster in stages.
Checklist
- Component mapping agreed
- Data copied and compacted; checksums/counts verified
- Glue catalog populated; Lake Formation permissions mapped from Ranger
- Jobs on EMR Serverless/transient clusters with S3-optimised committers
- Parallel run passed; Hadoop retired