Transformation: from hand-made notebook jobs to Databricks Asset Bundles and CI/CD

#Target workflow

Git branch → personal dev target (bundle deploy -t dev) → PR + CI (validate, tests) → merge → deploy to test → approval → deploy to prod

Nobody edits production jobs or notebooks in the UI.

#1. Start a bundle

databricks bundle init default-python       # template with src/, resources/, tests/

#2. Pull existing jobs into the bundle

databricks bundle generate job --existing-job-id 123456789 --profile prod

This writes resources/<job>.yml and downloads the notebooks/files it references into src/. Review and parameterise: cluster settings, paths, catalog names.

To make the bundle take over the existing job instead of creating a duplicate:

databricks bundle deployment bind sales_daily 123456789 -t prod

#3. Targets and variables

bundle:
  name: sales

variables:
  catalog: { default: dev }

targets:
  dev:
    mode: development          # prefixes resource names with [dev user], pauses schedules
    default: true
    workspace: { host: https://adb-111.11.azuredatabricks.net }
  test:
    workspace: { host: https://adb-222.22.azuredatabricks.net }
    variables: { catalog: test }
  prod:
    mode: production
    workspace: { host: https://adb-333.33.azuredatabricks.net }
    variables: { catalog: prod }
    run_as: { service_principal_name: "8a7c1e2f-0000-1111-2222-333344445555" }

Use ${var.catalog} in job parameters and code so the same code runs in every environment.

#4. Tests

• Unit-test pure Python logic with pytest locally.
• Integration tests run a small job in the test target (databricks bundle run -t test integration_tests).

#5. CI/CD

jobs:
  deploy-prod:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - run: databricks bundle validate -t prod
      - run: databricks bundle deploy -t prod
        env:
          DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST_PROD }}
          DATABRICKS_CLIENT_ID: ${{ vars.DATABRICKS_SP_ID }}
          DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_SP_SECRET }}

Prefer federation (OIDC) over a client secret where your setup supports it.

#6. Lock down production

• Production jobs owned and run by the deployment service principal.
• Users get CAN_VIEW / CAN_MANAGE_RUN on production jobs, not CAN_MANAGE.
• Production workspace folders read-only for people.

Warning

After binding, any manual change to the job in the UI is overwritten at the next deploy. Announce the switch so people stop editing in place.

#Checklist

• Bundle repo with generated/bound jobs
• Targets with variables for catalogs and hosts
• Tests in CI; validate on every PR
• Deployments by service principal
• UI edit rights removed in production