Home / Databricks on Azure

Databricks on AzureHow-toIntermediate

Advisory: Git folders and branching strategy for Databricks teams

Teams get into trouble when notebooks are edited in place in production or several people work on the same branch in one Git folder. Use personal Git folders on feature branches, keep code in files and wheels where sensible, choose a simple branching model, and deploy only through CI with Asset Bundles.

Deepak Bhardwaj2 min read

Applies toAzure Databricks Git folders (Repos) with Azure DevOps or GitHub, Databricks Asset Bundles, Databricks CLI

Principles

  • Git is the source of truth. The workspace is a working copy.
  • Every developer works in their own Git folder (under their user folder), on their own branch.
  • Production runs code deployed by CI, never code edited in a workspace.

Branching models

Model How Fits
Trunk-based Short feature branches → main; main deploys to test then prod with approvals Most teams; simple
Release branches main → release/x.y for prod; hotfixes on release Regulated or slower release cycles
Environment branches (dev/test/prod) A branch per environment Avoid; leads to drift and merge pain

Code layout

repo/
├─ databricks.yml            # bundle definition, targets dev/test/prod
├─ resources/                # jobs and pipelines as YAML
├─ src/
│  ├─ my_package/            # Python modules (tested, packaged as a wheel)
│  └─ notebooks/             # thin notebooks that call the package
├─ tests/                    # unit tests run in CI
└─ .github/workflows/ or azure-pipelines.yml

Keep heavy logic in Python files/modules with unit tests; notebooks become entry points and exploration.

CI/CD flow

  1. Pull request → lint and unit tests; databricks bundle validate.
  2. Merge to main → databricks bundle deploy -t test, run integration job.
  3. Approval → databricks bundle deploy -t prod by a service principal (OAuth / workload identity federation).
databricks bundle validate -t test
databricks bundle deploy -t test
databricks bundle run -t test integration_tests

Workspace settings that help

  • Restrict who can edit production folders; production jobs reference deployed bundle paths.
  • Use Git folder admin settings to limit allowed Git providers/URLs if needed.
  • Don't use Git folders under a shared path as the production code source.

Common anti-patterns

Anti-pattern Fix
Shared Git folder where several people commit Personal Git folders per developer
Hotfix edited directly in prod workspace Branch, PR, deploy via pipeline
Secrets in notebooks committed to Git Secret scopes/Key Vault; secret scanning in CI
Jobs pointing at a Git folder in a user's home Bundle-deployed paths, owned by a service principal

Tip

Start new projects from databricks bundle init templates so layout, targets and CI hooks are consistent across teams.

Checklist

  • Personal Git folders per developer
  • Branching model agreed (trunk-based by default)
  • Logic in tested modules; notebooks as entry points
  • CI/CD with bundles and a service principal
  • Production folders locked down
  • Secret scanning in CI

Related fixes