Install Python packages from a private Azure Artifacts feed on Azure Databricks: clusters, jobs and serverless

#Goal

Your team publishes internal Python packages to an Azure Artifacts feed. You want notebooks, jobs and serverless compute on Azure Databricks to install them with a plain %pip install or a job dependency. You don't want a personal access token pasted into every notebook.

There are three ways to do it. Use them in this order:

#Before you start: build the feed URL and token

The PyPI index URL for an Azure Artifacts feed looks like this:

# Project-scoped feed
https://pkgs.dev.azure.com/<org>/<project>/_packaging/<feed>/pypi/simple/

# Organization-scoped feed
https://pkgs.dev.azure.com/<org>/_packaging/<feed>/pypi/simple/

Create an Azure DevOps personal access token with the Packaging (Read) scope. Use a token that belongs to a service account, not a person, and note its expiry date. pip puts the token in the URL as the password, and the username can be any non-empty string:

https://build:<PAT>@pkgs.dev.azure.com/<org>/<project>/_packaging/<feed>/pypi/simple/

Test it from your laptop before you touch Databricks:

pip download mycompany-utils==1.4.2 --no-deps -d /tmp/check \
  --index-url "https://build:${ADO_PAT}@pkgs.dev.azure.com/contoso/data/_packaging/py-internal/pypi/simple/"

Tip

Turn on upstream sources (pypi.org) on the feed and use it as the only index. Then public packages also come through the feed. You don't need --extra-index-url https://pypi.org/simple/, and nobody can publish a public package with the same name as an internal one and get it installed by mistake.

#Option 1: workspace default package repositories

Workspace admins can store a default pip index URL in a reserved secret scope. Serverless notebooks, serverless jobs, classic clusters, Lakeflow pipelines and Databricks Apps then use it without any index flags in user code.

The scope name and key names are fixed. They must be exactly these:

# Create the reserved scope
databricks secrets create-scope databricks-package-management -p prod

# Admins manage it, everyone else can only read it
databricks secrets put-acl databricks-package-management admins MANAGE -p prod
databricks secrets put-acl databricks-package-management users READ -p prod

# Default index (the feed, with the token in the URL)
databricks secrets put-secret -p prod --json '{
  "scope": "databricks-package-management",
  "key": "pip-index-url",
  "string_value": "https://build:<PAT>@pkgs.dev.azure.com/contoso/data/_packaging/py-internal/pypi/simple/"
}'

Other keys you can set in the same scope:

• pip-extra-index-urls: more indexes, separated by spaces. Only add these if you can't use upstream sources.
• pip-cert: a CA certificate, for when a proxy or a private mirror uses an internal certificate authority.

You can also do this in the UI. Go to Settings > Compute > Default Package Repositories > Manage.

When the change takes effect:

Index settings in code (%pip install --index-url, a requirements.txt with --index-url, or an app's app.yaml) override the workspace default. If a team says the default "doesn't work", check their code for one of these first.

Warning

Users need READ on this scope. That means any user can read the token with dbutils.secrets.get. So give the token only Packaging (Read) on the feeds Databricks needs, never a broader scope.

#Option 2: %pip with a secret in a notebook

Put the token in a normal secret scope, read it in one cell, and use it in the next. %pip replaces $variable with the value of a Python variable:

# Cell 1
ado_pat = dbutils.secrets.get(scope="ado", key="artifacts-read-pat")

# Cell 2
%pip install mycompany-utils==1.4.2 --index-url https://build:$ado_pat@pkgs.dev.azure.com/contoso/data/_packaging/py-internal/pypi/simple/

Notebook output redacts secret values, but never print() the URL. Pin versions in production so that a new release in the feed doesn't change a job's behaviour overnight.

#Option 3: environment variable on a classic cluster

On classic compute you can set pip's own environment variables in the cluster's Advanced options > Spark > Environment variables, using a secret reference:

PIP_INDEX_URL={{secrets/ado/artifacts-index-url}}

Store the whole URL, token included, in the secret ado/artifacts-index-url. Cluster libraries and %pip on that cluster then use the feed. This doesn't apply to serverless, which has no cluster settings. Use Option 1 there.

#Declaring the dependency in a serverless job

For a serverless job task, list the package in the task's environment. With the default repository from Option 1, no index URL is needed. In a Databricks Asset Bundle:

resources:
  jobs:
    nightly_load:
      name: nightly_load
      environments:
        - environment_key: default
          spec:
            environment_version: "4"   # pick a version your workspace supports
            dependencies:
              - mycompany-utils==1.4.2
              - -r /Workspace/Shared/nightly_load/requirements.txt
      tasks:
        - task_key: load
          environment_key: default
          spark_python_task:
            python_file: ../src/load.py

Serverless environment versions change over time, so check which versions your workspace offers. In a serverless notebook, add the same lines in the Environment side panel under Dependencies, then click Apply.

#Troubleshooting

To check what pip actually sees, run this on the compute that fails:

%sh pip config list; env | grep -i '^PIP_' | sed 's/:[^@]*@/:***@/'

#Prevent it next time

• Set a calendar reminder or a pipeline alert before the PAT expires. An expired feed token breaks every job that installs a library when its cluster starts.
• Keep one feed with upstream sources as the only index, so internal names can't be taken over by public packages.
• Pin exact versions in job environments and requirements.txt files. Promote package versions through dev and prod the same way you promote code.
• Grant only admins MANAGE on databricks-package-management, and review who changes it in the audit logs.