Optimization: serverless jobs and notebooks on Azure Databricks - performance-optimized vs standard mode and budget policies

#Serverless modes for jobs

The mode is a job setting. Check current documentation for exact behaviour and pricing.

#When serverless tends to win

• Short, frequent jobs where classic cluster start time is a large share of the run.
• Spiky or unpredictable workloads.
• Teams without capacity to tune clusters.
• Avoiding idle all-purpose clusters for ad-hoc notebooks.

#When classic job compute may be cheaper

• Long, steady jobs that run for hours on well-tuned clusters.
• Jobs that benefit from spot VMs (not applicable to serverless).
• Workloads needing specific VM types, GPUs, init scripts or libraries that serverless doesn't support.

#Attribute and control spend

• Budget policies: attach a tag set to serverless usage so cost is attributed per team/project.
• Budgets: alerts when spend crosses thresholds.
• Workspace access: limit who can use serverless notebooks if needed.

-- Serverless spend by budget policy / tag over the last 30 days
SELECT usage_metadata.job_id,
       custom_tags['team'] AS team,
       billing_origin_product,
       SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE product_features.is_serverless = true
  AND usage_date >= current_date() - 30
GROUP BY ALL
ORDER BY dbus DESC;

#Interactive notebook hygiene

• Serverless notebooks bill while sessions are active; idle sessions end after a timeout.
• Long-running exploratory loops (while True, big display() of huge results) still cost.
• Point heavy recurring notebook work at scheduled jobs instead.

#Compare before switching

1. Pick 5 representative jobs.
2. Run each on serverless (both modes where relevant) and on tuned classic job compute.
3. Compare cost per run (serverless DBUs vs classic DBUs plus Azure VM cost) and run time.
4. Move the ones where serverless is cheaper or materially simpler.

Tip

Classic compute has two bills (DBUs and Azure VMs); serverless has one. Compare total cost per run, otherwise serverless looks more expensive than it is.

#Checklist

• Mode chosen per job (performance vs standard)
• Budget policies attached for attribution
• Budget alerts configured
• Notebook usage patterns reviewed
• Total cost per run compared with classic compute