Databricks Connect on Azure: run local Python against a cluster or serverless

#Goal

Write and debug PySpark code in your own IDE, with breakpoints, unit tests and Git, while the Spark work runs on Azure Databricks. Databricks Connect is a client library: your Python process runs locally, DataFrame operations are sent to a cluster or to serverless compute through Spark Connect, and results come back to your machine.

Use it when notebooks get too large to debug comfortably, or when you want pytest to run real Spark against real Unity Catalog tables.

#Requirements

• The workspace and account have Unity Catalog enabled.
• A classic cluster in dedicated or standard access mode, or serverless compute enabled for the workspace.
• A local Python whose minor version matches the compute. For Databricks Connect 16.4 and 17.3 (and serverless environment versions 3 and 4) that is Python 3.12; for 15.4 it is 3.11; for 13.3 and 14.3 it is 3.10. Check the "Databricks Connect versions" page for your release.
• The Databricks CLI, for OAuth sign-in.

Tip

The rule is: cluster Databricks Runtime version greater than or equal to the Databricks Connect package version. Pick the package that matches your cluster's runtime, for example 17.3.* for a 17.3 LTS cluster.

#1. Create a clean virtual environment

databricks-connect ships its own PySpark. A separately installed pyspark package conflicts with it and causes "stream corrupted" or "class not found" errors, so start clean.

python3.12 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip3 show pyspark && pip3 uninstall -y pyspark
pip3 install --upgrade "databricks-connect==17.3.*"

With Poetry:

poetry remove pyspark
poetry add databricks-connect@~17.3

Use ==X.Y.*, not ==X.Y, so you get the latest patch for that runtime.

Warning

On Windows, install into a path without spaces. A path like C:\Users\First Last\project fails with The filename, directory name, or volume label syntax is incorrect.

#2. Authenticate with OAuth

Sign in once with the CLI. It opens a browser and stores an OAuth profile in ~/.databrickscfg (%USERPROFILE%\.databrickscfg on Windows).

databricks auth login --host https://adb-1234567890123456.7.azuredatabricks.net --profile dev
databricks auth profiles

Find the cluster ID you want to use:

databricks clusters list --profile dev

Then add the compute target to the profile. Use either cluster_id or serverless_compute_id:

[dev]
host                  = https://adb-1234567890123456.7.azuredatabricks.net
auth_type             = databricks-cli
cluster_id            = 0123-456789-abcdefgh

[dev-serverless]
host                  = https://adb-1234567890123456.7.azuredatabricks.net
auth_type             = databricks-cli
serverless_compute_id = auto

Other supported options include Azure CLI sign-in (auth_type = azure-cli), an Entra ID service principal (azure_tenant_id, azure_client_id, azure_client_secret) and Azure managed identity (azure_use_msi, azure_client_id). Avoid personal access tokens for new setups.

#3. Create the session in code

Keep compute details out of the code and let the profile or environment decide:

from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.getOrCreate()

df = spark.read.table("samples.nyctaxi.trips")
df.groupBy("pickup_zip").count().orderBy("count", ascending=False).show(10)

Pick the target with environment variables:

export DATABRICKS_CONFIG_PROFILE=dev              # classic cluster
export DATABRICKS_SERVERLESS_COMPUTE_ID=auto      # optional: serverless; overrides cluster_id

In PowerShell use $env:DATABRICKS_CONFIG_PROFILE = "dev".

You can also be explicit while experimenting:

spark = DatabricksSession.builder.profile("dev").getOrCreate()
spark = DatabricksSession.builder.serverless().getOrCreate()
spark = DatabricksSession.builder.validateSession(True).getOrCreate()  # 14.3+

#How settings are resolved

Databricks Connect uses the first match in this order:

1. DatabricksSession.builder.remote(...) arguments
2. A profile named in code (.profile("dev"))
3. The DATABRICKS_CONFIG_PROFILE environment variable
4. Individual environment variables (DATABRICKS_HOST, DATABRICKS_CLUSTER_ID, ...)
5. The DEFAULT profile

If code connects to the wrong workspace, an old DATABRICKS_HOST or DEFAULT profile is usually the reason.

#4. Validate the setup

databricks-connect test

It exits non-zero with a message when the package, Python version or compute are incompatible. Run it in CI before the test suite.

#5. Use dbutils and write tests

dbutils.fs and dbutils.secrets are available locally through the Databricks SDK, which installs with Databricks Connect:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(profile="dev")
for f in w.dbutils.fs.ls("/Volumes/main/raw/landing/"):
    print(f.path, f.size)

A minimal pytest fixture:

# conftest.py
import pytest
from databricks.connect import DatabricksSession

@pytest.fixture(scope="session")
def spark():
    return DatabricksSession.builder.getOrCreate()

# test_transform.py
from mypkg.transform import clean_orders

def test_clean_orders_drops_null_ids(spark):
    df = spark.createDataFrame([(1, "a"), (None, "b")], "id INT, val STRING")
    assert clean_orders(df).count() == 1

In VS Code, the Databricks extension can manage the profile, cluster selection and Databricks Connect for you; the same profile file works either way.

#Common errors

#What doesn't work remotely

• display() on a DataFrame: use df.show() or df.limit(1000).toPandas().
• sparkContext, RDDs and libraries built on them (for example GraphFrames).
• dbutils.widgets, dbutils.notebook.run, dbutils.library and dbutils.credentials.
• CREATE TABLE ... AS SELECT through Databricks Connect: use spark.sql("SELECT ...").write.saveAsTable("main.sales.orders_clean").
• Distributed ML training.

#Prevent it next time

• Pin databricks-connect==X.Y.* in requirements.txt or pyproject.toml to the runtime your jobs use, and bump it in the same pull request as the job's runtime upgrade.
• Commit a .python-version file so everyone uses the same minor Python.
• Use serverless profiles for local testing so nobody waits on a cluster start.
• Run databricks-connect test as the first CI step and fail fast.
• Keep cluster_id and credentials out of code; use profiles locally and service principal environment variables in CI.