Applies toAzure Databricks with ADLS Gen2, Private Link / private endpoints, NAT gateway for secure cluster connectivity, Azure Cost Management
Where the extra cost appears
| Charge | Driver |
|---|---|
| ADLS transactions (read/write/list) | Many small files, frequent directory listing, chatty jobs |
| ADLS capacity | Delta history (time travel), unvacuumed files, raw data kept forever |
| Private endpoint data processing | Every GB through a private endpoint (storage, Event Hubs, Key Vault) |
| NAT gateway | Hours plus data processed for cluster egress (SCC with NAT) |
| Cross-region / internet egress | Storage in a different region from compute; downloads from the internet |
| Managed disks on VMs | Cluster disks per node |
Group Azure Cost Management by resource and meter for the storage accounts, private endpoints and NAT gateways used by Databricks.
Cut storage transactions
| Action | Effect |
|---|---|
| Compact small files (OPTIMIZE, auto compaction, optimized writes) | Fewer read operations per query |
| Auto Loader file notification mode for big landing zones | Avoids repeated directory listing (see the Auto Loader tuning post) |
Avoid dbutils.fs.ls loops over large trees in jobs | Listing is billed per operation |
| Delta log checkpoints in place (default) | Fewer log file reads |
Cut capacity
- Run
VACUUMwith a retention that matches your recovery needs. - Review
delta.logRetentionDurationanddelta.deletedFileRetentionDurationon large, high-churn tables. - Lifecycle rules for raw landing files once ingested (see the lake storage cost post).
Network paths
| Path | Cheaper option to check |
|---|---|
| Cluster → ADLS over private endpoint | Service endpoints for storage from the Databricks subnets (no PE processing charge) where your security policy allows |
| Cluster → internet via NAT | Mirror PyPI/Maven internally; avoid repeated large downloads in init scripts |
| Cross-region reads | Co-locate storage with the workspace; replicate rather than query across regions |
Private endpoints vs service endpoints is a security design choice too (see the Azure private link vs service endpoints guide). Don't trade away a required control for cost.
Find the busiest tables
-- Tables with many small files (candidates for compaction)
DESCRIBE DETAIL silver.web_events; -- numFiles, sizeInBytes
Large numFiles with small average size usually means high transaction cost too.
Tip
Tag storage accounts, NAT gateways and private endpoints with the same team/platform tags as the workspace, so these costs show up next to DBUs in reports.
Checklist
- Storage, PE, NAT and egress costs identified per meter
- Small files compacted; listing reduced
- VACUUM and retention reviewed
- Network paths reviewed against security policy
- Compute and storage in the same region
- Supporting resources tagged