Advisory: a well-architected review for Azure Databricks - questions and evidence queries per pillar

#Data and AI governance

• Are all workspaces on Unity Catalog, with Hive metastore usage retired?
• Are grants made to groups, not users?
• Are sensitive columns tagged and protected?

SELECT grantee, COUNT(*) FROM system.information_schema.table_privileges
WHERE grantee LIKE '%@%' GROUP BY grantee ORDER BY 2 DESC;   -- direct user grants

#Security

• VNet injection, secure cluster connectivity and Private Link in place?
• Serverless network access controlled (NCC, egress policies)?
• Personal access tokens disabled or restricted; OAuth for automation?
• Audit logs collected and detections in place?

#Reliability

• Production jobs on job compute/serverless with retries and alerts?
• DR plan for data, metadata and code, tested?
• Streaming checkpoints and Delta history protected?

-- Production jobs running on all-purpose clusters (last 30 days)
SELECT DISTINCT usage_metadata.job_id, usage_metadata.cluster_id
FROM system.billing.usage
WHERE usage_metadata.job_id IS NOT NULL
  AND sku_name LIKE '%ALL_PURPOSE%'
  AND usage_date >= current_date() - 30;

#Performance efficiency

• Large tables clustered and maintained (predictive optimization)?
• Photon and VM choices tested per heavy job?
• Slow queries and jobs tracked with owners?

#Cost optimization

• Cost attributed by tags and budget policies?
• Auto-termination on all-purpose clusters; idle compute tracked?
• Commitments sized to a stable baseline?

-- Untagged spend share (last 30 days)
SELECT SUM(CASE WHEN custom_tags['team'] IS NULL THEN usage_quantity END) / SUM(usage_quantity) AS untagged_share
FROM system.billing.usage
WHERE usage_date >= current_date() - 30;

#Operational excellence

• All jobs and pipelines deployed by Asset Bundles/CI?
• Runtimes on supported LTS versions with an upgrade schedule?
• Ops dashboard and alerting from system tables?

#Interoperability and usability

• Data accessible to other engines through open formats/sharing where needed?
• Self-service via SQL warehouses, Genie or BI with certified data?

#Running it

Tip

Save the evidence queries as a dashboard. The next review then starts from live numbers instead of a fresh data collection exercise.

#Checklist

• Evidence queries run
• All pillars covered
• Risks rated
• Top actions owned and dated
• Evidence dashboard kept for follow-up