Databricks Asset Bundles basics: write your first bundle, then validate, deploy and run it

#The goal

Someone built an important job in the UI. Nobody knows who changed it last, and copying it to production means clicking through the same screens again. With a bundle, the job is a YAML file in Git, and one command deploys it to any environment.

This post builds the smallest useful bundle: one notebook, one job and two targets.

#Step 1: install and log in

You need the newer CLI, not the old pip package:

databricks --version
databricks auth login --host https://adb-1234567890123456.7.azuredatabricks.net
databricks current-user me

If databricks --version shows 0.18 or lower, you're on the legacy CLI. Follow the installation steps in the CLI cheat sheet.

#Step 2: create the files

mkdir -p my_bundle/src
cd my_bundle

Create a notebook-style Python file at src/hello.py:

# Databricks notebook source
dbutils.widgets.text("env", "dev")
env = dbutils.widgets.get("env")

count = spark.range(100).count()
print(f"Environment: {env}, rows: {count}")

The first line makes it a notebook in the workspace. Now create databricks.yml in the bundle's root folder:

bundle:
  name: hello_bundle

variables:
  env_name:
    description: Environment passed to the job
    default: dev

resources:
  jobs:
    hello_job:
      name: hello_job
      tasks:
        - task_key: hello
          notebook_task:
            notebook_path: ./src/hello.py
            base_parameters:
              env: ${var.env_name}
          job_cluster_key: small
      job_clusters:
        - job_cluster_key: small
          new_cluster:
            spark_version: <lts-runtime-key>
            node_type_id: Standard_D4ds_v5
            num_workers: 1
            data_security_mode: SINGLE_USER

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://adb-1234567890123456.7.azuredatabricks.net

  prod:
    mode: production
    variables:
      env_name: prod
    workspace:
      host: https://adb-9999999999999999.9.azuredatabricks.net
      root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}

Replace <lts-runtime-key> with a runtime from databricks clusters spark-versions. To start from a template instead, run databricks bundle init, and choose default-python.

#Step 3: validate

databricks bundle validate

This checks the YAML, variables and references. It catches most typos without changing anything. To see the fully resolved config:

databricks bundle validate --output json

#Step 4: deploy

databricks bundle deploy -t dev
databricks bundle summary -t dev

summary lists what the bundle manages, with links. The files go to a folder under your user, and the job is created in the workspace.

Development mode makes it safe to experiment:

mode: production does extra checks, and it doesn't add prefixes. So deploy to production only through a controlled path.

#Step 5: run the job

Use the resource key from your YAML, not the job's display name:

databricks bundle run -t dev hello_job

The command waits for the run, prints a link to it and exits with a failure code if the job fails. That makes it useful in CI.

#Step 6: change something and redeploy

Edit src/hello.py, or add a schedule in databricks.yml:

    hello_job:
      name: hello_job
      schedule:
        quartz_cron_expression: "0 0 6 * * ?"
        timezone_id: "Asia/Kolkata"
        pause_status: UNPAUSED

Then run validate and deploy again. The bundle compares your files with what it deployed before, and updates only what has changed.

Warning

Once a bundle owns a job, treat the YAML as the truth. Changes made by hand in the UI can be overwritten by the next deploy. If someone needs to fix something fast, put the same change into Git straight afterward.

#Step 7: keep state and secrets out of Git

Add this to .gitignore:

.databricks/

That folder holds local state and cached data. Don't put tokens or passwords in databricks.yml. Use secret scopes at runtime.

#Step 8: deploy to production from CI

Use a service principal in your pipeline, rather than a person. Set the run identity in the prod target:

  prod:
    mode: production
    workspace:
      host: https://adb-9999999999999999.9.azuredatabricks.net
      root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}
    run_as:
      service_principal_name: "00000000-0000-0000-0000-000000000000"

A job owned by a person fails when that person leaves. The pipeline's commands are:

databricks bundle validate -t prod
databricks bundle deploy -t prod

Authenticate the pipeline with OAuth, not a personal token. See PAT to OAuth.

#Common errors

If a production deployment goes wrong, see bad bundle deploy rollback.

#Clean up

databricks bundle destroy -t dev

Caution

destroy deletes the jobs and other resources the bundle manages in that target, and its deployed files. Read the prompt, and be careful about which target you name. Running it against prod removes your production jobs.

#Prevent it next time

• Keep each bundle in Git, and review changes through pull requests.
• Deploy to production only from CI, with a service principal.
• Use one environment per stage, with targets for each.
• If you have existing jobs, see the migration to bundles. Read about Git folders for branching.