Azure Databricks costs missing tags in Azure Cost Management - cluster, pool, workspace and serverless tag propagation

#Symptoms

• In Azure cost analysis, grouping by your costcenter or team tag shows a large untagged bucket for the Databricks managed resource group.
• A cluster has the right custom tags in the Databricks UI, but its VMs and disks in Azure don't.
• A tag you added to the workspace last week still isn't on running clusters.
• Serverless notebooks, jobs or pipelines don't appear in Azure cost analysis by tag at all.
• A cluster or pool fails to start with an invalid settings error after a compute policy change.

#How Databricks tags reach Azure

The VM charges in your subscription carry the tags on the VMs. DBU charges are a separate meter. Look at a DBU line in your cost export to see which tags it carries before you build a chargeback report on it. For DBUs, system.billing.usage is the source to trust.

#Quick diagnosis

#Checks

#1. Is the cluster on a pool?

databricks clusters get 0612-091522-abcd1234 --output json \
  | jq '{name: .cluster_name, pool: .instance_pool_id, driver_pool: .driver_instance_pool_id, tags: .custom_tags}'

If pool or driver_pool has a value, the cluster's custom_tags never reach the VMs. Check the pool instead:

databricks instance-pools get 0612-085511-pool42 --output json | jq '.custom_tags'

#2. What tags do the VMs actually have?

Find the managed resource group and list VM tags:

MRG_ID=$(az databricks workspace show -g rg-data-prod -n dbw-data-prod \
  --query managedResourceGroupId -o tsv)
MRG=${MRG_ID##*/}

az vm list -g "$MRG" --query "[].{vm:name, cluster:tags.ClusterName, costcenter:tags.costcenter, pool:tags.DatabricksInstancePoolId}" -o table

VMs with a pool value and no costcenter confirm the pool problem. VMs with no costcenter and no pool mean the cluster wasn't tagged or hasn't been restarted since tagging.

#3. What tags does the workspace have?

az databricks workspace show -g rg-data-prod -n dbw-data-prod --query tags

Add or change workspace tags with a merge, so existing tags stay:

WS_ID=$(az databricks workspace show -g rg-data-prod -n dbw-data-prod --query id -o tsv)
az tag update --resource-id "$WS_ID" --operation Merge --tags costcenter=CC1234 env=prod

Warning

Don't try to tag the managed resource group or its resources directly. Databricks manages them, they are protected by a deny assignment, and Databricks sets VM tags at launch. Tag the workspace, pools and clusters instead.

#4. How much DBU spend is untagged?

Use the billing system table, which covers classic and serverless:

SELECT
  billing_origin_product,
  usage_metadata.cluster_id   AS cluster_id,
  usage_metadata.job_id       AS job_id,
  usage_metadata.warehouse_id AS warehouse_id,
  identity_metadata.run_as    AS run_as,
  SUM(usage_quantity)         AS dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - INTERVAL 30 DAYS
  AND custom_tags['costcenter'] IS NULL
GROUP BY ALL
ORDER BY dbus DESC
LIMIT 50;

Fix the top rows first: usually a few shared all-purpose clusters, pools or untagged serverless jobs account for most of it.

#Fixes

#Tag pools, not just clusters

If teams share a pool, cluster tags are lost on the VMs. Create one pool per cost owner, or accept pool-level allocation:

databricks instance-pools edit --json '{
  "instance_pool_id": "0612-085511-pool42",
  "instance_pool_name": "pool-finance-ds4",
  "node_type_id": "Standard_D4ds_v5",
  "custom_tags": {"costcenter": "CC1234", "team": "finance"}
}'

The edit call takes the full pool definition. Copy the current settings from instance-pools get first so you don't reset the idle and size limits. Existing idle instances keep their old tags, and new tags apply as the pool expands.

#Enforce tags with compute policies

Make the tag required and limited to known values:

{
  "custom_tags.costcenter": { "type": "allowlist", "values": ["CC1234", "CC2001", "CC3005"] },
  "custom_tags.team":       { "type": "unlimited", "isOptional": false }
}

Apply it to the policy used by each team's clusters, then restart running clusters so the tags reach the VMs.

Caution

Never add a policy tag whose key matches a default tag (Vendor, Creator, ClusterName, ClusterId, JobId, RunName) or a pool tag. Clusters under that policy fail to start, including production jobs.

#Tag serverless with usage policies

Serverless usage policies (Public Preview, formerly called budget policies) tag serverless notebooks, jobs, pipelines and model serving endpoints. Create a policy per team in the workspace settings, add the custom tags and assign users or groups. Those tags then show up in system.billing.usage.custom_tags, not in Azure cost analysis.

#Fill gaps in Azure reports

For the Azure side, turn on tag inheritance in Cost Management settings. It applies subscription and resource group tags to usage records, so untagged Databricks VM costs at least pick up the subscription's cost center. It is a reporting change and doesn't modify resource tags.

#Prevent it next time

• Treat workspace tags as the baseline (costcenter, env, owner), and use pool and cluster tags only for finer splits.
• Require tags through compute policies for classic compute and usage policies for serverless.
• Schedule the untagged-DBU query weekly as a SQL alert to the FinOps channel.
• Build chargeback from system.billing.usage joined to system.billing.list_prices for DBUs, and from Azure cost exports for VMs, disks and networking. Don't expect one source to cover both.
• Keep the total tag count per resource well under 50 after aggregation.