Advanced Databricks Jobs: for each tasks, if/else conditions, task values and job parameters

Databricks Jobs grew from "run this notebook at 02:00" into a capable orchestrator. For pipelines that live entirely in Databricks, these features often remove the need for Airflow or ADF around them.

#Job parameters and dynamic references

Job-level parameters are pushed to every task:

parameters:
  - name: run_date
    default: "{{job.start_time.iso_date}}"
  - name: env
    default: dev

Useful dynamic value references:

#Task values: pass data between tasks

Upstream notebook:

files = [f.path for f in dbutils.fs.ls(f"/Volumes/sales_prod/raw/landing/{run_date}/")]
dbutils.jobs.taskValues.set(key="file_count", value=len(files))
dbutils.jobs.taskValues.set(key="files", value=files)

Downstream notebook:

count = dbutils.jobs.taskValues.get(taskKey="discover", key="file_count", default=0, debugValue=0)

Task values must be JSON-serializable and small (the docs give the current size limit); pass paths, not data.

#If/else condition tasks

- task_key: has_files
  depends_on: [{ task_key: discover }]
  condition_task:
    op: GREATER_THAN
    left: "{{tasks.discover.values.file_count}}"
    right: "0"

- task_key: load
  depends_on: [{ task_key: has_files, outcome: "true" }]
  notebook_task: { notebook_path: ../notebooks/load.py }

- task_key: notify_no_files
  depends_on: [{ task_key: has_files, outcome: "false" }]
  notebook_task: { notebook_path: ../notebooks/notify_no_files.py }

Operators include EQUAL_TO, NOT_EQUAL, GREATER_THAN, GREATER_THAN_OR_EQUAL, LESS_THAN, LESS_THAN_OR_EQUAL. Comparisons are on strings or numbers as the values allow.

#For each tasks

Run the same nested task once per input, with a concurrency limit:

- task_key: load_each_file
  depends_on: [{ task_key: has_files, outcome: "true" }]
  for_each_task:
    inputs: "{{tasks.discover.values.files}}"
    concurrency: 8
    task:
      task_key: load_one
      notebook_task:
        notebook_path: ../notebooks/load_one.py
        base_parameters:
          path: "{{input}}"
      max_retries: 2

• inputs is a JSON array (literal or a task value reference). Objects work too; refer to fields with {{input.field}}.
• Each iteration has its own retries and logs; the for each task fails if any iteration fails after retries.
• There is a maximum number of iterations per for each task; check the current limit and batch inputs if you exceed it.

Warning

Each iteration on classic compute shares the job cluster. With concurrency: 8, size the cluster for eight concurrent loads, or use serverless. Too much concurrency against a source database causes throttling, not speed.

#Run-if rules: cleanup and alerts

- task_key: cleanup
  depends_on: [{ task_key: load_each_file }, { task_key: notify_no_files }]
  run_if: AT_LEAST_ONE_SUCCESS
  notebook_task: { notebook_path: ../notebooks/cleanup.py }

- task_key: on_failure_ticket
  depends_on: [{ task_key: load_each_file }]
  run_if: AT_LEAST_ONE_FAILED
  notebook_task: { notebook_path: ../notebooks/open_ticket.py }

Options: ALL_SUCCESS (default), AT_LEAST_ONE_SUCCESS, NONE_FAILED, ALL_DONE, AT_LEAST_ONE_FAILED, ALL_FAILED. Combined with condition tasks, "skipped" branches count as neither success nor failure, so test the rule you choose.

#The whole job as a bundle

# resources/ingest_job.yml
resources:
  jobs:
    sales_ingest:
      name: sales-ingest-${bundle.target}
      max_concurrent_runs: 1
      parameters:
        - { name: run_date, default: "{{job.start_time.iso_date}}" }
      tasks:
        - task_key: discover
          notebook_task: { notebook_path: ../notebooks/discover.py }
        # has_files, load_each_file, notify_no_files, cleanup as above
      email_notifications:
        on_failure: [data-oncall@contoso.com]

databricks bundle validate -t prod
databricks bundle deploy -t prod
databricks bundle run sales_ingest -t prod --params run_date=2026-10-28

#Repair and backfill

• Repair run reruns only failed tasks (and failed for each iterations), keeping task values from successful upstream tasks.
• For backfills, trigger runs with explicit run_date parameters, or use the backfill option in the jobs UI where available.

databricks jobs run-now <job-id> --json '{"job_parameters": {"run_date": "2026-10-20"}}'

#When to stay with an external orchestrator

#Prevent it next time

• Concurrency on every for each task set deliberately.
• Failure-path tasks (AT_LEAST_ONE_FAILED) for tickets, not just emails.
• Jobs defined in bundles, deployed by CI, never edited only in the UI.