Applies toAzure Databricks serverless (jobs, notebooks, pipelines, SQL), classic job and all-purpose clusters, SQL warehouses (serverless, pro, classic), instance pools
Workload → compute
| Workload | First choice | Alternative |
|---|---|---|
| Scheduled production jobs | Serverless jobs | Classic job compute (tuned, spot) |
| Declarative pipelines | Serverless pipelines | Classic pipelines compute |
| BI dashboards, SQL analysts, Power BI | Serverless SQL warehouse | Pro warehouse where serverless isn't available or allowed |
| Interactive data engineering / data science | Serverless notebooks | Shared all-purpose cluster with policy and auto-termination |
| GPU ML training | Classic GPU clusters | Managed ML services where they fit |
| Jobs needing custom images, init scripts, special libraries | Classic job compute |
Classic still matters when
- You need specific VM families, GPUs, local SSD sizes or spot VMs.
- Init scripts, custom containers or OS-level packages are required.
- Network rules require compute in your own VNet with paths that serverless can't use yet.
- Specific Spark configurations aren't supported on serverless.
Check current serverless limitations; the list shrinks with each release.
Access modes on classic clusters
| Mode | Use |
|---|---|
| Standard (formerly shared) | Multi-user, Unity Catalog enforced, most workloads |
| Dedicated (formerly single user) | One user or group; needed for some ML/RDD/GPU features |
Avoid no-isolation clusters; they bypass Unity Catalog controls.
Guardrails
| Guardrail | How |
|---|---|
| No production on all-purpose | Jobs must use job compute or serverless (policy + review) |
| Limited cluster creation | Cluster policies; most users get serverless or a policy-bound personal compute |
| Cost attribution | Tags via policies; budget policies for serverless |
| Auto-termination | Enforced on all-purpose clusters |
Decision record per workload
Note the compute type, why (latency, features, cost per run), owner and review date. Revisit when serverless gains the missing feature.
Tip
Make serverless the default and require a short justification for classic compute. That flips the conversation from "why serverless?" to "what do you need that serverless doesn't give?".
Checklist
- Workloads mapped to compute types
- Classic exceptions justified
- Access modes chosen; no-isolation removed
- Policies enforce job compute for production
- Tags and budget policies applied