Databricks can't import your own Python module: ModuleNotFoundError with workspace files, Git folders and sys.path

#Symptom

A notebook does from utils.cleaning import normalise and fails with:

ModuleNotFoundError: No module named 'utils'

Variations of the same problem:

• It works when you run the notebook by hand but fails when a job runs it.
• It works for you but not for a colleague who cloned the same repo.
• The import works, but your latest edit to the .py file is ignored.
• The import works on the driver, but a UDF that calls the module fails on executors.
• A well-known package (for example requests or logging) suddenly behaves strangely or raises AttributeError.

All of these come down to what is on sys.path, what the file actually is, and which copy Python already loaded.

#How Databricks builds the Python path

• On Databricks Runtime 11.3 LTS and above, the notebook's current working directory is added to sys.path automatically. Modules next to the notebook import with no setup.
• In a Git folder, the repo root is added as well, and it is placed before installed libraries. Outside Git folders, the working directory is added after installed libraries.
• Anything you add yourself with sys.path.append() goes to the end of the path and has the lowest precedence.
• On Runtime 13.3 LTS and above, directories on sys.path and directories structured as Python packages are distributed to executors. On 12.2 LTS and below they are not, so UDFs that import your code fail on workers.

#Step 1: print what Python sees

Run this in a cell at the top of the failing notebook, and add it temporarily to the job:

import os, sys
print("cwd:", os.getcwd())
print("runtime:", spark.conf.get("spark.databricks.clusterUsageTags.sparkVersion", "unknown"))
for p in sys.path:
    print(p)

Things to look for:

• cwd is not the folder you expected (a job running from a different copy, or another user's folder).
• The folder that contains utils/ is missing from the list.
• Paths written as /Repos/... or /Users/... without the /Workspace prefix.

#Step 2: confirm the module is a file, not a notebook

Only workspace files can be imported. A notebook cannot, even if it has a .py extension in Git. A .py file whose first line is # Databricks notebook source is imported into the workspace as a notebook.

Check the object type with the Databricks CLI:

databricks workspace get-status /Workspace/Users/someone@contoso.com/project/utils/cleaning.py
databricks workspace list /Workspace/Users/someone@contoso.com/project/utils

If object_type is NOTEBOOK, either remove the # Databricks notebook source header from the file in Git and pull again, or keep it as a notebook and use %run ./utils/cleaning instead of import.

Tip

Add an empty __init__.py to each folder you import from. Namespace packages work without it in plain Python, but an explicit package makes behaviour predictable across runtimes and executors.

#Step 3: match the cause

#Step 4: apply the fix

#Module in another folder

Build the path relative to the notebook so it works in every clone:

import os, sys
repo_src = os.path.abspath(os.path.join(os.getcwd(), "..", "src"))
if repo_src not in sys.path:
    sys.path.append(repo_src)

from utils.cleaning import normalise

If you must use an absolute path, include /Workspace:

sys.path.append("/Workspace/Shared/data-eng/common/src")

Omitting /Workspace from a Git folder path causes the import to fail.

#Edits not picked up

Python caches imported modules for the life of the interpreter. In interactive work, turn on autoreload at the top of the notebook:

%load_ext autoreload
%autoreload 2

For a one-off, reload the module explicitly:

import importlib, utils.cleaning
importlib.reload(utils.cleaning)

Or restart the Python process, which also clears notebook-scoped state:

dbutils.library.restartPython()

Jobs start a fresh process for each run, so this only affects interactive sessions. If a job still runs old code, it is reading an old copy. Check the commit or path the run used.

#Name shadowing

In a Git folder the repo root comes first on sys.path, so a file called logging.py or a folder called requests/ hides the installed package. Find the culprit:

import requests
print(requests.__file__)   # should be under site-packages, not your repo

Rename the module and delete any __pycache__ folder beside it.

#Imports in UDFs

On Runtime 13.3 LTS and above, a module imported from a directory on sys.path is available to executors. Import it at the top of the notebook, not only inside a driver-side function, and make sure the folder is a package. On older runtimes, or when many jobs share the code, build a wheel and install it as a library.

#Jobs and Git folders

• A job task that points at a workspace path runs whatever is currently in that folder. If someone switched the Git folder to a feature branch, the job imports the feature branch code.
• A job with a Git source checks out the configured branch, tag or commit for each run, so imports resolve against that checkout.
• Lakeflow Declarative Pipelines append the pipeline root folder to sys.path. Put shared modules under that root or append the path explicitly in the pipeline source.

Check what the job points at:

databricks jobs get <job-id> --output json | jq '{git_source: .settings.git_source, tasks: [.settings.tasks[] | {task_key, notebook: .notebook_task.notebook_path, source: .notebook_task.source}]}'

Warning

Never point production jobs at a personal Git folder or a folder people switch branches in. A branch switch silently changes the code the next run imports. Use Git source with a tag or commit, or deploy with Databricks Asset Bundles.

#Prevent it next time

• Keep shared code in a src/ package with __init__.py files, and build sys.path entries relative to the notebook, never from a user's home folder.
• Do not name modules after standard library or installed packages.
• Remove # Databricks notebook source from files that are meant to be imported.
• Use Runtime 13.3 LTS or later so imported modules reach executors.
• For code shared by many jobs, package it as a wheel and version it, rather than relying on path tricks.
• Add a smoke test task to CI that imports every module on the target runtime before deploying.