Applies toDatabricks on Azure, AWS and GCP with Unity Catalog system tables enabled
1. Where does the money go?
-- list-price cost by SKU for the last 30 days
SELECT u.sku_name,
ROUND(SUM(u.usage_quantity * lp.pricing.default), 0) AS list_cost_usd
FROM system.billing.usage u
JOIN system.billing.list_prices lp
ON u.sku_name = lp.sku_name
AND u.usage_start_time >= lp.price_start_time
AND (lp.price_end_time IS NULL OR u.usage_start_time < lp.price_end_time)
WHERE u.usage_date >= current_date() - 30
GROUP BY u.sku_name
ORDER BY list_cost_usd DESC;
List prices ignore your discounts, but they're fine for relative comparisons. Cloud VM costs (classic compute) are billed separately by Azure/AWS.
2. Attribute to jobs, warehouses and teams
SELECT u.usage_metadata.job_id, j.name,
ROUND(SUM(u.usage_quantity), 1) AS dbus
FROM system.billing.usage u
LEFT JOIN (SELECT workspace_id, job_id, name FROM system.lakeflow.jobs
QUALIFY ROW_NUMBER() OVER (PARTITION BY workspace_id, job_id ORDER BY change_time DESC) = 1) j
ON u.usage_metadata.job_id = j.job_id AND u.workspace_id = j.workspace_id
WHERE u.usage_date >= current_date() - 30 AND u.usage_metadata.job_id IS NOT NULL
GROUP BY ALL ORDER BY dbus DESC LIMIT 20;
system.lakeflow.jobs keeps a row per change to a job, so take the latest row per job (as above) to avoid double counting.
-- by custom tag (team / cost centre)
SELECT u.custom_tags['team'] AS team, ROUND(SUM(u.usage_quantity), 0) AS dbus
FROM system.billing.usage u
WHERE u.usage_date >= current_date() - 30
GROUP BY ALL ORDER BY dbus DESC;
Untagged spend is your first finding — enforce tags with cluster policies and budget policies (for serverless).
3. The levers
| Lever | Typical saving | How |
|---|---|---|
| Jobs on job compute, not all-purpose clusters | Large (job compute DBU rate is much lower) | Move scheduled notebooks to jobs with job clusters |
| Auto-termination on all-purpose clusters | Idle hours | Policy: autotermination_minutes fixed at 30–60 |
| Right-size clusters | 10–30% | Check CPU/memory in system.compute.node_timeline; fewer/smaller workers |
| Spot / preemptible workers with fallback | 30–60% on VM cost | SPOT_WITH_FALLBACK, driver on-demand |
| Photon | Depends | Faster on SQL/DataFrame-heavy work; higher DBU rate — benchmark per job |
| Serverless jobs/SQL | Removes idle and start-up waste | Good for short, spiky jobs and BI |
| SQL warehouse auto-stop | Idle hours | 5–10 min for serverless; scale min clusters to 1 |
| Table optimisation | Less compute per query | Liquid clustering, predictive optimisation (see that post) |
| Kill stale objects | Varies | Unused jobs, pools with idle instances, forgotten clusters |
4. Find idle and oversized compute
SELECT cluster_id, ROUND(AVG(cpu_user_percent + cpu_system_percent), 1) AS avg_cpu,
ROUND(AVG(mem_used_percent), 1) AS avg_mem
FROM system.compute.node_timeline
WHERE start_time >= current_date() - 14
GROUP BY cluster_id
HAVING avg_cpu < 20
ORDER BY avg_cpu;
5. Guardrails
- Cluster policies for every user group (see the policies post).
- Budgets with alerts in the account console; serverless budget policies for tag attribution.
- Monthly cost review per team using a dashboard on the queries above.
Tip
The single biggest saving we see in most estates: production notebooks scheduled on all-purpose clusters that stay up all day. Moving them to job compute with auto-scaling often cuts that spend by more than half.