Sev1: an IP access list change locked everyone out of the Databricks workspace

#Symptoms

• Browser, CLI and REST calls return HTTP 403 with a message like: Source IP address: x.x.x.x is blocked by Databricks IP ACL for workspace: 1234567890123456.
• BI tools, ADF linked services and external schedulers fail at the same time.
• Jobs already running inside Databricks keep going (IP access lists apply to front-end access), but nothing new can be triggered from outside.

#Step 1: find a path back in (5 minutes)

IP access lists check the public source IP seen by Databricks.

1. Get your egress IP: curl -s https://checkip.amazonaws.com.
2. Try from any network that is in the current allow list: office network, a specific VPN profile, a jump host, a build agent whose IP is listed.
3. Clients that reach the workspace through front-end Private Link are not evaluated against public IP lists in the same way; check the current documentation, but a VM on a VNet with private access is often a way in.

From an allowed host:

databricks auth login --host https://adb-1234567890123456.7.azuredatabricks.net
databricks ip-access-lists list -o json | jq -r '.[] | [.list_id, .label, .list_type, .enabled, (.ip_addresses|join(","))] | @tsv'
databricks workspace-conf get-status enableIpAccessLists

#Step 2: fix or disable

Fix the list (add the missing ranges):

databricks ip-access-lists update <list-id> --json '{
  "label": "corp-egress",
  "list_type": "ALLOW",
  "enabled": true,
  "ip_addresses": ["203.0.113.0/24", "198.51.100.17/32", "192.0.2.64/27"]
}'

Or, for fastest relief, disable the feature until the list is right:

databricks workspace-conf set-status --json '{"enableIpAccessLists": "false"}'

Warning

Disabling IP access lists reopens the workspace to any IP (authentication still applies). Only do it if your security policy allows, and re-enable the same day.

Changes can take a few minutes to apply.

#Step 3: nobody can reach the workspace

If no allowed network is reachable:

• An account admin can manage workspace network settings from the account console in some configurations; check whether the account console offers IP access list or network configuration for the workspace.
• Use a Private Link path if one exists (a VM in the transit VNet).
• Otherwise, open a Severity A case with Microsoft Azure support for Azure Databricks from the Azure portal (the support blade is outside the workspace and not affected by the IP ACL). Have the workspace ID, region, time of change and desired ranges ready.

#Step 4: collect every client's egress IP

The workspace audit logs help you find who was connecting from where before the change:

SELECT source_ip_address, user_identity.email, count(*) AS calls, max(event_time) AS last_seen
FROM system.access.audit
WHERE workspace_id = 1234567890123456 AND event_time > current_timestamp() - INTERVAL 7 DAYS
GROUP BY ALL
ORDER BY calls DESC;

Blocked requests are logged with action names related to IP access denial; filter recent events with response.status_code = 403 to see which clients are still blocked after the fix.

#Step 5: confirm

• Users can open the workspace from office and VPN.
• Each integration (ADF, Power BI, CI) succeeds a test call.
• enableIpAccessLists back to true if you disabled it.

#Rolling out IP access lists safely

1. Gather all egress ranges (step 4) and add a BLOCK list only for known bad ranges if needed.
2. Create the ALLOW list with "enabled": false first, review it with the network team.
3. Enable during working hours with two admins on different allowed networks.
4. Watch the audit log for 403s for an hour; add missed ranges.

databricks ip-access-lists create --json '{"label":"corp-egress","list_type":"ALLOW","ip_addresses":["203.0.113.0/24"]}'

#Prevent it next time

• Keep the allow list in Terraform (databricks_ip_access_list) with comments naming each range's owner.
• Always include a dedicated break-glass egress (an admin jump host with a fixed public IP) in the list.
• Prefer Private Link for corporate access and keep public access minimal.
• Alert on changes to IP access lists via the audit log.