Databricks Jobs basics: schedule a notebook, add retries, parameters and failure alerts

A notebook that runs when you press Run is a draft. A job runs on a schedule, on the right compute, retries when something transient fails, and tells someone when it cannot recover.

#Step 1: make the notebook job-ready

• No hard-coded dates: read a parameter.
• Idempotent: running it twice for the same date gives the same result (MERGE or overwrite a partition, not plain append).
• No display() of huge results.

dbutils.widgets.text("run_date", "")
run_date = dbutils.widgets.get("run_date") or spark.sql("select date_sub(current_date(), 1)").first()[0]

(spark.read.table("sales_dev.raw.orders")
   .filter(f"order_date = '{run_date}'")
   .write.mode("overwrite")
   .option("replaceWhere", f"order_date = '{run_date}'")
   .saveAsTable("sales_dev.silver.orders"))

#Step 2: create the job

Jobs & Pipelines > Create > Job:

• Task type: Notebook, source Git (preferred) or Workspace.
• Compute: Serverless if available, otherwise a job cluster (new for each run). Avoid all-purpose clusters for scheduled work.
• Parameters: run_date = {{job.start_time.iso_date}} or leave empty to use the default logic.

#Step 3: schedule

Schedules & Triggers > Add trigger > Scheduled. Pick the timezone explicitly; UTC avoids daylight-saving surprises. Quartz cron example for 02:30 every day: 0 30 2 * * ?.

Other triggers: File arrival (when new files land in a volume or external location) and Continuous.

#Step 4: retries and timeouts

On the task:

• Retries: 2, with a minimum interval of a few minutes. Covers transient cloud and network errors.
• Timeout: a value well above the normal duration (for example 3x). A hung job then fails instead of running all night.
• Duration warning threshold: get a warning when the job runs longer than usual, before it times out.

Warning

Retries re-run the whole task. If the notebook appends data, a retry after a partial write creates duplicates. Make it idempotent first.

#Step 5: notifications

On the job: Job notifications > Add: email, Teams/Slack webhooks or PagerDuty destinations configured by admins. Select Failure at minimum; Duration warning is useful; skip Start and Success for most jobs or people will start ignoring the mails.

#Step 6: run as a service principal

By default a job runs as its creator. If that person leaves, the job breaks. Set Run as to a service principal that has the needed Unity Catalog grants:

GRANT USE CATALOG ON CATALOG sales_dev TO `sp-sales-etl`;
GRANT USE SCHEMA, SELECT, MODIFY ON SCHEMA sales_dev.silver TO `sp-sales-etl`;
GRANT USE SCHEMA, SELECT ON SCHEMA sales_dev.raw TO `sp-sales-etl`;

Changing "Run as" to a service principal requires the Service Principal User role on it.

#The same job as JSON

{
  "name": "sales-silver-orders-daily",
  "schedule": { "quartz_cron_expression": "0 30 2 * * ?", "timezone_id": "UTC", "pause_status": "UNPAUSED" },
  "max_concurrent_runs": 1,
  "timeout_seconds": 7200,
  "email_notifications": { "on_failure": ["data-oncall@contoso.com"] },
  "tasks": [{
    "task_key": "silver_orders",
    "notebook_task": { "notebook_path": "notebooks/silver_orders", "source": "GIT",
                       "base_parameters": { "run_date": "" } },
    "max_retries": 2,
    "min_retry_interval_millis": 300000,
    "timeout_seconds": 3600
  }],
  "git_source": { "git_url": "https://dev.azure.com/contoso/data/_git/sales", "git_provider": "azureDevOpsServices", "git_branch": "main" }
}

databricks jobs create --json @job.json
databricks jobs run-now <job-id>
databricks jobs list-runs --job-id <job-id> --limit 5

With no compute block, this definition uses serverless jobs compute where it is enabled. For a classic job cluster, add a job_clusters entry and reference it from the task with job_cluster_key. For anything beyond a few jobs, define them in Databricks Asset Bundles so they live in Git and deploy through CI.

#Reading a failed run

Open the run, then the failed task:

1. The error at the top usually names the exception.
2. Output shows the notebook with the cell that failed.
3. For Spark errors, the driver logs and Spark UI (classic compute) show more.

Repair a run to rerun only failed tasks:

databricks jobs repair-run <run-id> --rerun-all-failed-tasks

#Prevent it next time

• max_concurrent_runs: 1 for jobs that must not overlap.
• Every production job: run as service principal, failure notifications, timeout, retries.
• Keep job definitions in Git (bundles) rather than editing them only in the UI.