Home / Databricks on Azure

Databricks on AzureRunbookIntermediate

Transformation: Synapse Spark pools and notebooks to Azure Databricks

Move Spark workloads from Azure Synapse Spark pools to Azure Databricks. Map pools to clusters, Synapse lake database tables to Unity Catalog, mssparkutils to dbutils, linked services to external locations, and pipelines to Workflows or ADF.

Deepak Bhardwaj2 min read

Applies toAzure Synapse Analytics Spark pools and notebooks, Synapse lake databases, Azure Databricks with Unity Catalog

Mapping

Synapse Databricks
Spark pool (size, autoscale, auto-pause) Job cluster / serverless jobs; cluster policies
Notebooks (.ipynb with Synapse metadata) Notebooks or Python files in Git folders/bundles
Lake database / Spark tables (Hive-compatible metastore) Unity Catalog catalogs/schemas/tables
mssparkutils dbutils (and Python libraries)
Linked services, workspace managed identity Storage credentials (access connector) + external locations
Synapse pipelines Databricks Workflows, or ADF pipelines calling Databricks
Packages (workspace/pool packages) Cluster/job libraries, wheels in volumes

1. Inventory

Export notebooks, Spark job definitions, pool configurations and lake database table definitions (from Synapse Studio or the Git repo connected to the workspace). Note Spark versions — Synapse runtimes (3.3/3.4) vs. Databricks Runtime versions.

2. Storage and tables

Data is usually already in ADLS Gen2 as Parquet/Delta. Register it in Unity Catalog rather than copying:

CREATE EXTERNAL LOCATION syn_lake URL 'abfss://lake@stsynapse.dfs.core.windows.net/' WITH (STORAGE CREDENTIAL sc_lake);
CREATE TABLE main.sales.orders USING DELTA LOCATION 'abfss://lake@stsynapse.dfs.core.windows.net/sales/orders';

Parquet tables can be converted with CONVERT TO DELTA.

3. Code changes

# Synapse
from notebookutils import mssparkutils
files = mssparkutils.fs.ls("abfss://lake@stsynapse.dfs.core.windows.net/raw/")
secret = mssparkutils.credentials.getSecret("kv-data", "api-key", "ls_keyvault")
mssparkutils.notebook.run("child_notebook", 600, {"date": "2026-10-25"})

# Databricks
files = dbutils.fs.ls("abfss://lake@stsynapse.dfs.core.windows.net/raw/")      # or /Volumes/... paths
secret = dbutils.secrets.get(scope="kv-data", key="api-key")                    # Key Vault-backed scope
dbutils.notebook.run("child_notebook", 600, {"date": "2026-10-25"})

Also check: %%configure magic blocks (move to cluster/job config), spark.read.synapsesql() connector usage (use the Azure Synapse/SQL connectors or JDBC), and table names (use three-level names).

4. Orchestration

Either rebuild Synapse pipelines as Databricks Workflows (bundles), or move pipelines to ADF and replace Notebook activities with Databricks Job activities.

5. Validate and cut over

Run jobs side by side writing to test schemas; compare outputs and runtimes. Switch consumers (Power BI, serverless SQL views) to Databricks SQL or keep reading the same Delta files. Pause and delete Spark pools when idle for the rollback window.

Tip

If Synapse serverless SQL views read the same Delta files, they keep working during the move — a useful way to keep reports alive while Spark jobs migrate.

Checklist

  • Notebooks, job definitions, pool configs exported
  • Storage registered in UC; tables mapped
  • mssparkutils, %%configure and connectors replaced
  • Pipelines rebuilt; parallel run reconciled
  • Spark pools removed

Related fixes