Sev1: Databricks jobs fail installing libraries — PyPI/Maven blocked, version conflicts, yanked packages

#Symptoms

• Job runs fail with Library installation failed, Could not find a version that satisfies the requirement, or connection timeouts to pypi.org / repo1.maven.org.
• Jobs worked yesterday with no code change — a dependency released a new version overnight.

#1. Read the library status

databricks libraries cluster-status <cluster-id> --output json | jq '.library_statuses[] | {library, status, messages}'

For job clusters, the run's error and the cluster's driver log (stderr) show the pip/maven output.

#2. Get tonight's jobs running

• Pin to the last known good versions (from a previous successful run's library list or your lock file) and re-run:

"libraries": [ { "pypi": { "package": "pandas==2.2.2" } }, { "pypi": { "package": "great-expectations==0.18.19" } } ]

• Egress blocked: allow pypi.org and files.pythonhosted.org (and Maven repos) in the firewall, or point to your mirror:

{ "pypi": { "package": "requests==2.32.3", "repo": "https://pkgs.dev.azure.com/contoso/_packaging/pypi-mirror/pypi/simple/" } }

• Private feed down: install from wheels stored in a Unity Catalog volume:

{ "whl": "/Volumes/ops/libs/wheels/contoso_utils-1.4.2-py3-none-any.whl" }

#3. Lock dependencies properly

Keep a requirements.txt generated from a lock (pip-tools or uv) with exact versions including transitive dependencies, stored with the job code:

uv pip compile requirements.in -o requirements.txt

Reference it from the job (serverless environments and cluster libraries both support requirements files in workspace files or volumes):

{ "requirements": "/Volumes/ops/libs/orders-job/requirements.txt" }

#4. Package your own code as a wheel

Build internal code as a wheel in CI, version it, upload to a volume or Azure Artifacts, and reference the exact version — instead of %pip install git+https://... at runtime.

#5. Use the runtime's libraries where possible

Databricks Runtime ships many libraries pre-installed. Installing a different version of a core library (e.g. numpy, pyarrow) can conflict with the runtime. Check the runtime release notes for included versions and only override when needed.

Tip

Test library changes on the target runtime version in a dev job before merging. Upgrading the Databricks Runtime and libraries in the same change makes failures hard to diagnose.

#Prevent it next time

• Exact pins with a lock file; no unpinned installs in production jobs.
• An internal mirror (Azure Artifacts upstream sources) with allow-listed packages.
• Notebook-scoped %pip install only in development.