Onboarding a new team to Azure Databricks: identity, catalog, compute, folders and cost checklist

#Goal

A new team (here, finance-analytics) needs to work in an existing Azure Databricks workspace by the end of the day. You want the same result every time. The team should be able to sign in, see only its own data, start compute you approve, schedule jobs that don't run as a person, and show up correctly in the cost reports.

Work through the sections in order. Each step has a check, so you can confirm it worked before you hand over.

#Before you start

Collect these from the requester. Don't create anything until you have all of them.

Tip

Keep this as a template ticket or a YAML file in Git. Once onboarding works from a file, you can automate it later with Terraform or bundles.

#1. Identity

• Group exists at account level. With automatic identity management or SCIM provisioning from Entra ID, the Entra group appears in the account console once it's synced. Don't create a separate workspace-local group with the same name.
• Group assigned to the workspace with the USER permission (not ADMIN).

# find the group's ID in the account
databricks account groups list --filter 'displayName eq "sg-dbx-finance-analytics"'

# assign it to the workspace (workspace ID from the workspace URL or Azure portal)
databricks account workspace-assignment update <workspace-id> <group-id> \
  --json '{"permissions": ["USER"]}'

• Entitlements kept minimal. Give Workspace access and Databricks SQL access. Leave out Allow unrestricted cluster creation, because compute should come through policies (section 3).

Check: a team member signs in and sees the workspace, but can't create a cluster without picking a policy.

#2. Data in Unity Catalog

Give the team its own schema to write to, and read access to the shared data the owner approved.

-- team-owned output schema
CREATE SCHEMA IF NOT EXISTS prod.finance_analytics
  COMMENT 'Owned by finance-analytics. Contact: data owner in ticket DBX-1234';
ALTER SCHEMA prod.finance_analytics OWNER TO `sg-dbx-finance-analytics-admins`;

GRANT USE CATALOG ON CATALOG prod TO `sg-dbx-finance-analytics`;
GRANT USE SCHEMA, SELECT, MODIFY, CREATE TABLE, CREATE VIEW
  ON SCHEMA prod.finance_analytics TO `sg-dbx-finance-analytics`;

-- read-only access to approved shared data
GRANT USE SCHEMA, SELECT ON SCHEMA prod.sales TO `sg-dbx-finance-analytics`;

• Ownership sits with a group, never an individual, so the schema survives people leaving.
• No grants on the whole catalog beyond USE CATALOG.
• If the team needs to land files, create a volume in its schema rather than granting access to a storage path:

CREATE VOLUME IF NOT EXISTS prod.finance_analytics.landing;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME prod.finance_analytics.landing TO `sg-dbx-finance-analytics`;

Check:

SHOW GRANTS `sg-dbx-finance-analytics` ON SCHEMA prod.sales;
SHOW GRANTS ON SCHEMA prod.finance_analytics;

Warning

Granting SELECT at schema level includes tables added to that schema later. If prod.sales will hold sensitive tables, grant on specific tables, or protect columns with masks, instead of the whole schema.

#3. Compute

#Cluster policy

Reuse an existing policy where possible. Only create a team-specific one if the team needs different limits, and set the cost tags in it (section 6).

databricks cluster-policies list
databricks cluster-policies update-permissions <policy-id> --json '{
  "access_control_list": [
    {"group_name": "sg-dbx-finance-analytics", "permission_level": "CAN_USE"}
  ]
}'

Caution

set-permissions replaces every direct permission on the object. Running it with only the new team in the list removes access for every other team using that policy, and their jobs will fail. Use update-permissions to add, and check with get-permissions afterwards.

#SQL warehouse

For SQL and dashboard users, give CAN_USE on a shared warehouse rather than creating one per team:

databricks warehouses list
databricks warehouses update-permissions <warehouse-id> --json '{
  "access_control_list": [
    {"group_name": "sg-dbx-finance-analytics", "permission_level": "CAN_USE"}
  ]
}'

• Team can start compute only through the approved policy.
• Auto-termination is enforced by the policy, not left to users.

Check: databricks cluster-policies get-permissions <policy-id> lists both the existing groups and the new one.

#4. Workspace folders and Git

databricks workspace mkdirs /Workspace/Teams/finance-analytics
databricks workspace get-status /Workspace/Teams/finance-analytics   # note object_id

databricks workspace update-permissions directories <object-id> --json '{
  "access_control_list": [
    {"group_name": "sg-dbx-finance-analytics", "permission_level": "CAN_MANAGE"}
  ]
}'

• Shared code lives in a team folder or Git folder, not in one person's home folder.
• Production jobs point at a Git repository or a deployed bundle, not at notebooks someone is editing.
• Each user sets up their own Git credentials (Settings > Linked accounts). Don't share one person's token.

#5. A service principal for scheduled jobs

Jobs must not run as a team member. When that person leaves, the jobs break.

databricks account service-principals create --display-name "sp-dbx-finance-analytics-jobs"
databricks account workspace-assignment update <workspace-id> <sp-id> \
  --json '{"permissions": ["USER"]}'

GRANT USE CATALOG ON CATALOG prod TO `<sp-application-id>`;
GRANT USE SCHEMA, SELECT, MODIFY ON SCHEMA prod.finance_analytics TO `<sp-application-id>`;
GRANT USE SCHEMA, SELECT ON SCHEMA prod.sales TO `<sp-application-id>`;

• Give the team group the Service Principal: User role on the service principal so the team can set it as a job's run-as identity (the role name and location in the UI vary by version).
• Prefer OAuth or a managed identity for any external caller. Don't create personal access tokens for the service principal unless a tool requires one, and set an expiry if you do.

Check: create a small test job with Run as set to the service principal, and confirm it can read prod.sales and write to prod.finance_analytics.

#6. Cost attribution

• The cluster policy sets fixed custom tags, for example team = finance-analytics and cost_centre = CC-4410. Users can't override them.
• If the team uses serverless, it's assigned a budget policy carrying the same tags.
• A budget alert exists in the account console for the team's tag (Databricks usage) and, if needed, in Azure Cost Management for the VM side.

Check after the team's first day of usage:

SELECT usage_date, sku_name, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE custom_tags['team'] = 'finance-analytics'
  AND usage_date >= current_date() - INTERVAL 7 DAYS
GROUP BY ALL
ORDER BY usage_date;

No rows after real usage means the tags aren't being applied. Check the policy before the month-end report goes out with the cost unassigned.

#7. Hand-over

• One team member signs in and runs SELECT current_user(), is_account_group_member('sg-dbx-finance-analytics'). It should return true.
• They query a table in prod.sales, then try writing to a schema they shouldn't have and confirm it fails.
• Send the team a short note: their schema, warehouse, policy name, folder, service principal and who approves new data access.
• Record what you created (groups, schema, volume, policy, folder, service principal, tags) in the ticket.

#Prevent drift next time

• Put sections 2 to 6 in Terraform or a script that takes the team name and cost centre as inputs. Manual onboarding drifts after a handful of teams.
• Review grants each quarter: SHOW GRANTS ON SCHEMA for each team schema and system.access.audit for who is actually querying what.
• Use the same checklist in reverse when a team is disbanded: move job ownership, remove grants, drop or archive the schema, and remove the workspace assignment last.