Home / Data Platform / Governance

Data PlatformHow-toIntermediate

Advisory: cost allocation and showback for shared data platforms

Shared warehouses, clusters and capacities make it hard to see who drives cost. Choose allocation keys per platform, tag or separate compute by team, split shared costs fairly, start with showback before any chargeback, and use the numbers to drive optimisation rather than arguments.

Deepak Bhardwaj3 min read

Applies toDatabricks (system billing tables, tags), Snowflake (warehouses, query attribution), Microsoft Fabric capacities, cloud cost tools (Azure Cost Management, AWS Cost Explorer/CUR)

Why allocate

  • Teams change behaviour when they can see their own cost.
  • Platform budgets are easier to defend with clear drivers.
  • Optimisation work can target the biggest consumers.

Allocation keys by platform

Platform Direct attribution Shared costs to split
Databricks Tags on clusters, jobs, SQL warehouses, serverless budget policies; system billing tables Shared all-purpose clusters, platform jobs
Snowflake Warehouse per team, query tags, query attribution views Shared warehouses, storage, serverless features
Fabric Capacity per domain or workspace usage from capacity metrics Shared capacity
Cloud storage/network Tags per container/account Shared lake zones, egress 
-- Databricks: cost by team tag for last month (list price; adjust for your discounts)
SELECT u.custom_tags['team'] AS team,
       SUM(u.usage_quantity * p.pricing.default) AS est_cost
FROM system.billing.usage u
JOIN system.billing.list_prices p
  ON u.sku_name = p.sku_name
 AND u.usage_start_time >= p.price_start_time
 AND (p.price_end_time IS NULL OR u.usage_start_time < p.price_end_time)
WHERE u.usage_date >= date_trunc('month', add_months(current_date(), -1))
  AND u.usage_date <  date_trunc('month', current_date())
GROUP BY 1 ORDER BY 2 DESC;

-- Snowflake: credits by query tag (attribution view; check availability on your edition)
SELECT query_tag, SUM(credits_attributed_compute) AS credits
FROM snowflake.account_usage.query_attribution_history
WHERE start_time >= DATEADD(month, -1, CURRENT_DATE)
GROUP BY 1 ORDER BY 2 DESC;

Design choices

Choice Options
Separate compute or shared + tags Separate warehouses/clusters per team is clearest; shared with tags is cheaper but needs query-level attribution
Shared cost split Proportional to direct usage, fixed share, or held centrally as platform cost
Untagged usage Policies that require tags; report untagged as its own line and drive it to near zero
Granularity Team or domain level first; per-product later if needed

Showback before chargeback

  1. Showback: monthly report per team, no money moves. Agree the method.
  2. Fix tagging gaps and disputed allocations.
  3. Chargeback (optional): move budgets only once teams trust the numbers.

Make it useful

  • Pair the cost report with top drivers (biggest jobs, warehouses, queries) and suggested actions.
  • Show unit costs where possible (cost per pipeline run, per dashboard, per TB processed).
  • Review trends monthly with team leads.

Tip

Enforce tags through cluster policies, budget policies and warehouse creation rules. Tagging that relies on people remembering will drift within weeks.

Checklist

  • Allocation keys chosen per platform
  • Tags enforced by policy
  • Shared cost split method agreed
  • Showback report running monthly
  • Untagged usage tracked and reduced
  • Drivers and unit costs included

Related fixes