Home / Databricks on Azure

Databricks on AzureHow-toAdvanced

Advisory: networking for Databricks serverless on Azure - network connectivity configs, private endpoints and egress control

Serverless compute runs in Databricks' account, not your VNet, so classic VNet injection rules don't apply. Use network connectivity configurations for stable egress identities and private endpoints to your resources, control outbound access with serverless network policies, and design firewall rules on storage and other services around them.

Deepak Bhardwaj2 min read

Applies toAzure Databricks serverless compute (SQL warehouses, jobs, notebooks, pipelines), network connectivity configurations (NCC), serverless egress control / network policies; check regional availability

Classic vs serverless networking

Classic compute Serverless compute
Where it runs Your VNet (VNet injection) Databricks-managed network in your region
Reaching your storage Private endpoints/service endpoints from your subnets NCC: stable subnet IDs for firewall rules, or private endpoints created via NCC
Outbound control Your NSG, firewall, NAT Serverless egress control (network policies)
Front-end access Private Link front-end, IP access lists Same workspace front-end controls

Network connectivity configurations (NCC)

An NCC is an account-level object attached to workspaces. It provides:

  • Stable service subnets you can allow on storage account firewalls.
  • Private endpoint rules so serverless reaches your ADLS, and other supported resources, privately. You approve the endpoint on the resource side.
# Illustrative: create an NCC and a private endpoint rule with the account-level CLI
databricks account network-connectivity create-network-connectivity-configuration \
  --json '{"name":"ncc-prod-centralindia","region":"centralindia"}'

databricks account network-connectivity create-private-endpoint-rule <ncc-id> \
  --json '{"resource_id":"/subscriptions/<sub>/resourceGroups/rg-data/providers/Microsoft.Storage/storageAccounts/stlakeprod","group_id":"dfs"}'

Then approve the pending private endpoint connection on the storage account and attach the NCC to the workspace. Check exact CLI syntax for your CLI version.

Egress control

Serverless network policies can restrict outbound internet access to allowed destinations (e.g. specific package repositories or APIs), with logging of denied attempts. Use them to meet "no uncontrolled egress" rules.

Design choices

Question Option
One NCC per region or per environment? Often per region and environment; NCCs are regional
Firewall rules or private endpoints to storage? Private endpoints for strict "no public endpoint" policies; subnet allow-listing is simpler
Which resources need private access? ADLS, Key Vault, SQL sources, other data stores; check supported resource types
How to allow package installs? Egress policy allow-list or internal mirrors

Rollout

  1. Inventory what serverless workloads must reach.
  2. Create NCC(s) and private endpoint rules; approve on resources.
  3. Attach to workspaces; test from a serverless notebook and SQL warehouse.
  4. Apply egress policies in dry-run/log mode if available, then enforce.

Warning

Turning on storage firewall rules before serverless access is configured breaks SQL warehouses and serverless jobs immediately. Add NCC rules and test first.

Checklist

  • Serverless destinations inventoried
  • NCC per region/environment created
  • Private endpoints approved or subnets allow-listed
  • Egress policies designed and enforced
  • Tested from each serverless compute type

Related fixes