Azure Databricks standard access mode errors: RDD, SparkContext, UDF and Spark config failures

#Symptoms

A notebook or job that worked on a dedicated (single user) or legacy no-isolation cluster fails after it moves to a cluster in standard access mode, often during a Unity Catalog migration or after a cluster policy change. Typical errors:

• [UNSUPPORTED_CONNECT_FEATURE.RDD] Feature is not supported in Spark Connect: Resilient Distributed Datasets (RDDs) are not available on Standard access mode or Serverless compute.
• [UNSUPPORTED_CONNECT_FEATURE.SESSION_SPARK_CONTEXT] ... SparkContext is not available on Standard access mode or Serverless compute.
• Cluster create or edit fails because a Spark config is restricted.
• Init scripts or JAR/Maven libraries are rejected because they are not on the allowlist.
• Code that calls the Azure instance metadata service (IMDS) to get a managed identity token hangs or is refused.

Standard access mode runs user code through Spark Connect as a low-privilege user so several users can share one cluster under Unity Catalog. Anything that reaches into the JVM, the driver's SparkContext or the host is blocked by design. The fix is either to rewrite the code to the DataFrame API or to run that workload on dedicated compute.

#1. Confirm the access mode and runtime

The exact limits depend on the Databricks Runtime version, so get both values first.

databricks clusters get 0123-456789-abcdefgh --output json \
  | jq '{name: .cluster_name, mode: .data_security_mode, dbr: .spark_version}'

USER_ISOLATION (or DATA_SECURITY_MODE_STANDARD) is standard access mode. SINGLE_USER (or DATA_SECURITY_MODE_DEDICATED) is dedicated. Names vary by API version; check your CLI's output.

To find every active cluster in standard mode across the workspace, use the compute system table:

SELECT cluster_id, cluster_name, owned_by, data_security_mode, dbr_version
FROM system.compute.clusters
WHERE delete_time IS NULL
QUALIFY ROW_NUMBER() OVER (PARTITION BY cluster_id ORDER BY change_time DESC) = 1
ORDER BY data_security_mode, cluster_name;

#2. Map the error to the cause

Older runtimes have more gaps. For example, Python scalar and pandas UDFs need 13.3 LTS, applyInPandas and mapInPandas need 14.3 LTS, and Spark ML needs 17.0. If the code is valid but still fails, check the runtime-dependent table on the "Standard compute requirements and limitations" page for your version before rewriting anything.

#3. Rewrite the common patterns

Most failures come from a short list of RDD and SparkContext calls. These replacements work on standard, dedicated and serverless compute, so the code becomes portable.

#Building small DataFrames

# Before: sc.parallelize([(1, "a"), (2, "b")]).toDF(["id", "val"])
df = spark.createDataFrame([(1, "a"), (2, "b")], "id INT, val STRING")

Rows passed to spark.createDataFrame from local data must stay under 128 MB each.

#Reading raw files

# Before: sc.textFile("abfss://raw@stacct.dfs.core.windows.net/logs/")
lines = spark.read.text("/Volumes/main/raw/logs/")

# Before: sc.wholeTextFiles(path)
files = spark.read.format("binaryFile").load("/Volumes/main/raw/docs/")

Replace input_file_name() with the _metadata column:

from pyspark.sql import functions as F
df = spark.read.json("/Volumes/main/raw/events/").select("*", F.col("_metadata.file_path").alias("src_file"))

#Row-level logic that used rdd.map

# Before: df.rdd.map(lambda r: (r.id, parse(r.payload))).toDF()
from pyspark.sql.types import StringType

@F.udf(StringType())
def parse_udf(payload):
    return parse(payload)

out = df.withColumn("parsed", parse_udf("payload"))

For batch-oriented logic, mapInPandas is usually faster than a row UDF on DBR 14.3 LTS and above.

#Configuration and logging

# Before: spark.sparkContext.getConf().get("spark.sql.shuffle.partitions")
spark.conf.get("spark.sql.shuffle.partitions")

# Before: spark.sparkContext.setLogLevel("WARN")  -> remove; use Python logging
import logging
logging.getLogger("myjob").setLevel(logging.WARNING)

sc.broadcast variables have no direct equivalent. For joins, use a broadcast hint: big.join(F.broadcast(small), "key").

#Hive UDFs to Unity Catalog functions

CREATE OR REPLACE FUNCTION main.ops.normalize_sku(sku STRING)
RETURNS STRING
LANGUAGE PYTHON
AS $$
  return sku.strip().upper() if sku else None
$$;

GRANT EXECUTE ON FUNCTION main.ops.normalize_sku TO `data-engineers`;

Caution

Python foreachBatch on standard access mode does not support ThreadPoolExecutor or other multi-threaded execution inside the function. Databricks warns this can produce data corruption or inconsistent results without an error. Remove threading from foreachBatch before moving a stream to standard mode.

#4. Allowlist init scripts and JAR/Maven libraries

On standard access mode, init scripts and JAR or Maven libraries must match the Unity Catalog artifact allowlist. A metastore admin, or a user with MANAGE_ALLOWLIST, maintains it.

databricks artifact-allowlists get INIT_SCRIPT
databricks artifact-allowlists get LIBRARY_MAVEN

Warning

update replaces the entire list for that artifact type. Export the current list first and merge your new entry, or you will break every other cluster that depended on the old entries.

databricks artifact-allowlists get LIBRARY_MAVEN --output json > maven-allowlist.json
# edit the file: keep existing artifact_matchers, append the new one
databricks artifact-allowlists update LIBRARY_MAVEN --json @maven-allowlist.json

Example entry for an init script folder in a volume:

{"artifact_matchers": [{"artifact": "/Volumes/ops/platform/init-scripts/", "match_type": "PREFIX_MATCH"}]}

Allowlist a narrow prefix (a specific volume folder or Maven group), not a whole catalog or a broad path.

#5. When to stop rewriting and use dedicated compute

Standard mode is the recommended default for classic compute, but some workloads belong on dedicated compute:

A dedicated cluster can be assigned to a group rather than a single user, so a team can share it. For scheduled jobs, use job compute in dedicated mode with the job's service principal as the run-as identity.

#Prevent it next time

• Test notebooks on standard access mode or serverless before a migration, not after the cluster policy changes.
• Add a lint rule in CI that flags .rdd, sparkContext, sc. and setLogLevel in new code.
• Keep the artifact allowlist in source control and apply it with Terraform or the CLI from a merged file, so update never drops entries.
• Pin each job's runtime and record which runtime-dependent features it uses, then check the limitations page when you upgrade.
• Run the system table query from section 1 monthly and review clusters still on legacy no-isolation mode.