Databricks Asset Bundles across dev, test and prod: variables, lookups, presets and run_as done properly

#Goal

You have one repository and three workspaces, and your first bundle works. Now the YAML is full of copy-pasted cluster specs and hard-coded warehouse IDs, and dev deployments sometimes start schedules that should stay paused. This post shows a layout that scales: one set of resource definitions, per-target differences kept in variables and presets, and production runs that never depend on a human identity.

Note

Databricks has started to rename Asset Bundles to Declarative Automation Bundles in its documentation. The CLI commands (databricks bundle ...) and the databricks.yml format are the same.

#Layout

databricks.yml
resources/
  ingest.job.yml
  gold.job.yml
  silver.pipeline.yml
src/
  ...

# databricks.yml
bundle:
  name: sales_lakehouse

include:
  - resources/*.yml

variables:
  catalog:
    description: Target Unity Catalog catalog
  warehouse_id:
    description: SQL warehouse for SQL tasks
    lookup:
      warehouse: "shared-etl"
  etl_policy_id:
    lookup:
      cluster_policy: "etl-job-compute"
  job_cluster:
    description: Standard job cluster spec
    type: complex
    default:
      spark_version: 15.4.x-scala2.12
      node_type_id: Standard_D8ds_v5
      autoscale:
        min_workers: 2
        max_workers: 8
      data_security_mode: SINGLE_USER

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://adb-1111111111111111.11.azuredatabricks.net
    variables:
      catalog: dev

  test:
    workspace:
      host: https://adb-2222222222222222.12.azuredatabricks.net
      root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}
    presets:
      name_prefix: "[test] "
      trigger_pause_status: PAUSED
      tags:
        env: test
    variables:
      catalog: test
    run_as:
      service_principal_name: "00000000-0000-0000-0000-00000000aaaa"

  prod:
    mode: production
    workspace:
      host: https://adb-3333333333333333.13.azuredatabricks.net
      root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}
    variables:
      catalog: prod
      job_cluster:
        spark_version: 15.4.x-scala2.12
        node_type_id: Standard_E16ds_v5
        autoscale:
          min_workers: 4
          max_workers: 20
        data_security_mode: SINGLE_USER
    run_as:
      service_principal_name: "00000000-0000-0000-0000-00000000bbbb"
    permissions:
      - group_name: data-platform-ops
        level: CAN_MANAGE
      - group_name: sales-analysts
        level: CAN_VIEW

The runtime version, VM sizes and IDs above are examples. Use the LTS runtime and node types you have standardized on.

#Variables: plain, complex and lookup

Lookups stop the most common multi-workspace bug: a warehouse or policy ID copied from dev that doesn't exist in prod. Give the shared object the same name in every workspace, and the bundle resolves the right ID for each one.

A job that uses all three kinds:

# resources/gold.job.yml
resources:
  jobs:
    gold_daily:
      name: gold_daily
      max_concurrent_runs: 1
      schedule:
        quartz_cron_expression: "0 30 5 * * ?"
        timezone_id: UTC
      parameters:
        - name: catalog
          default: ${var.catalog}
      job_clusters:
        - job_cluster_key: main
          new_cluster:
            policy_id: ${var.etl_policy_id}
            spark_version: ${var.job_cluster.spark_version}
            node_type_id: ${var.job_cluster.node_type_id}
            autoscale: ${var.job_cluster.autoscale}
            data_security_mode: ${var.job_cluster.data_security_mode}
      tasks:
        - task_key: build_gold
          job_cluster_key: main
          notebook_task:
            notebook_path: ../src/gold/build_gold.py
        - task_key: refresh_marts
          depends_on: [{task_key: build_gold}]
          sql_task:
            warehouse_id: ${var.warehouse_id}
            file:
              path: ../src/gold/refresh_marts.sql

Tip

Reference whole sub-objects (${var.job_cluster.autoscale}) instead of repeating every field. Then a prod override of job_cluster changes all jobs at once.

#Modes and presets

mode: development prefixes resource names with [dev <user>], pauses all schedules and triggers, and deploys to the user's home folder. That's what you want on a shared dev workspace, because each engineer gets an isolated copy.

mode: production adds safety checks. Validation fails if, for example, the deployment would use a personal workspace path. Combine it with a shared root_path and run_as.

presets set the same kind of behaviour without a mode:

Precedence: presets override the mode's defaults, and settings on an individual resource override presets.

Warning

According to the bundle docs, a schedule set explicitly to UNPAUSED on the job wins over trigger_pause_status: PAUSED. If test must never run on a schedule, don't hard-code pause_status: UNPAUSED in the shared resource file. Set it per target instead.

#run_as: who the jobs execute as

The deploying identity (your CI service principal) and the run identity can differ:

• run_as.service_principal_name takes the application ID of the service principal.
• The deployer needs the Service principal: User role on that service principal, and the run-as service principal needs the Unity Catalog grants the jobs require.
• Pipelines support only some run_as combinations. If validation rejects one, check the current docs for your CLI version.

Check what will be deployed before CI does it:

databricks bundle validate -t prod
databricks bundle summary -t prod

summary lists resolved resource names, IDs and workspace paths. A lookup that resolves to the wrong object shows up here.

#Adopt jobs that already exist

Don't delete hand-made production jobs and recreate them. That loses run history and changes job IDs that orchestrators depend on. Generate YAML from the existing job, then bind it:

databricks bundle generate job --existing-job-id 123456 -t prod
# review and refactor resources/*.yml, replace literals with ${var.*}
databricks bundle deployment bind gold_daily 123456 -t prod
databricks bundle deploy -t prod

After binding, the bundle owns the job. Edits made in the UI are overwritten on the next deploy, so tell the team before you bind.

#CI pipeline sketch

databricks bundle validate -t test
databricks bundle deploy   -t test
databricks bundle run      -t test gold_daily --no-wait
# integration checks ...
databricks bundle validate -t prod
databricks bundle deploy   -t prod

Authenticate CI with an Entra ID service principal or Databricks OAuth M2M, not a personal access token. The CLI reads DATABRICKS_HOST plus either ARM_CLIENT_ID, ARM_CLIENT_SECRET and ARM_TENANT_ID, or DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET.

#Prevent it next time

• Fail the PR if databricks bundle validate fails for any target, not just dev.
• Keep IDs out of YAML. Use lookups, or ${resources.jobs.<key>.id} for references inside the bundle.
• Put max_concurrent_runs: 1 on jobs that aren't safe to run in parallel. A preset default doesn't replace a deliberate value.
• Restrict CAN_MANAGE on prod jobs to the deploy principal and an ops group, so UI drift can't happen quietly.
• Pin the CLI version in CI and upgrade it on purpose. Bundle schema fields are still being added.