Letting external engines read Unity Catalog tables on Azure Databricks: Iceberg REST catalog and credential vending checklist

#Goal

Another engine (open-source Spark, Trino, PyIceberg, Snowflake and others) needs to read Unity Catalog tables without its own storage keys and without bypassing Unity Catalog permissions. The supported route: the engine calls Unity Catalog, which checks privileges and vends short-lived ADLS credentials scoped to the table. It's straightforward when every item below is in place, and it fails with confusing 403s when one is missing.

Work through the sections in order. Every box should be ticked before you tell the consuming team it's ready.

#1. Decide what you are exposing

• List the exact tables. Expose a dedicated schema of curated tables (a "data product" schema), not whole catalogs.
• Confirm each table type is supported:

• No exposed table relies on row filters or column masks. Table credential vending doesn't support those tables, because the external engine reads files directly and can't enforce them. Publish a filtered copy, or keep those consumers on Databricks SQL.

Caution

Credential vending hands out storage access to the table's files. Anything you protect with row filters, column masks or a view's WHERE clause must not be exposed this way. Expose a separate table that contains only the permitted data.

#2. Make Delta tables readable as Iceberg

• For Delta tables, enable Iceberg reads (UniForm):

ALTER TABLE prod.share.orders SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

• Deletion vectors are disabled on those tables. Iceberg reads require it. To remove existing ones:

REORG TABLE prod.share.orders APPLY (UPGRADE UNIFORM(ICEBERG_COMPAT_VERSION=2));

• Writers to these tables run DBR 14.3 LTS or later.
• Verify with SHOW TBLPROPERTIES prod.share.orders; or the Delta Uniform Iceberg section of DESCRIBE EXTENDED.
• If Iceberg metadata lags behind the Delta commits, regenerate it: MSCK REPAIR TABLE prod.share.orders SYNC METADATA;

Warning

Column mapping and the protocol upgrade are one-way for practical purposes. Older Databricks runtimes and other Delta readers that don't support these features will stop reading the table. Check every existing reader, and test on a clone first.

#3. Enable external access on the metastore

• A metastore admin has turned on External data access: Catalog > gear icon > Metastore > Details. It's off by default.
• A change ticket records who enabled it and why. It's a metastore-wide setting, so audit and security teams should know.

#4. Identity and privileges

• The external engine authenticates as a dedicated service principal, not a user. Use OAuth machine-to-machine (client ID and secret), and keep personal access tokens out of it.
• Grants, applied explicitly by the catalog owner:

GRANT USE CATALOG ON CATALOG prod TO `sp-trino-reader`;
GRANT USE SCHEMA  ON SCHEMA  prod.share TO `sp-trino-reader`;
GRANT SELECT      ON SCHEMA  prod.share TO `sp-trino-reader`;
GRANT EXTERNAL USE SCHEMA ON SCHEMA prod.share TO `sp-trino-reader`;

• You understand that ALL PRIVILEGES does not include EXTERNAL USE SCHEMA, and schema owners don't get it automatically. This is the most common reason for "permission denied" when ordinary SELECT works fine inside Databricks.
• The service principal has workspace access to the workspace whose URL the engine uses, and that workspace is bound to the catalog if catalog isolation is on.

#5. Network paths (two of them)

The engine needs to reach both the workspace API and the storage account.

• Workspace URL reachable from the engine's network. Allow its egress IPs if IP access lists are on, or give it a route to the front-end Private Link endpoint.
• Storage reachable. Vended credentials are SAS tokens for https://<account>.dfs.core.windows.net. If the ADLS firewall allows only private endpoints and Databricks resource instances, the external engine also needs a private endpoint or an allowed network path to the storage account.
• DNS for privatelink.dfs.core.windows.net resolves correctly from the engine's network.

# From the engine host: both must resolve and connect
nslookup adb-<id>.<n>.azuredatabricks.net
nslookup <account>.dfs.core.windows.net
curl -sS -o /dev/null -w "%{http_code}\n" https://<account>.dfs.core.windows.net/

#6. Client configuration

• The engine uses the Iceberg REST endpoint https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest, with the Unity Catalog catalog name as warehouse.
• Iceberg library version isn't 1.9.0, where credential refresh doesn't work. Use a current release so long-running queries survive token expiry.

PyIceberg example:

catalog:
  unity_catalog:
    uri: https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest
    warehouse: prod
    token: <oauth-access-token>

Spark with OAuth:

spark.sql.catalog.uc=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.uc.type=rest
spark.sql.catalog.uc.uri=https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest
spark.sql.catalog.uc.warehouse=prod
spark.sql.catalog.uc.rest.auth.type=oauth2
spark.sql.catalog.uc.oauth2-server-uri=https://<workspace-url>/oidc/v1/token
spark.sql.catalog.uc.credential=<client-id>:<client-secret>
spark.sql.catalog.uc.scope=all-apis

Include org.apache.iceberg:iceberg-azure-bundle on non-Databricks clusters.

• Smoke test: load one table's metadata.

curl -sS -H "Authorization: Bearer $TOKEN" \
  "https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest/v1/catalogs/prod/namespaces/share/tables/orders" \
  | head -c 500

A 403 here is a privilege or metastore-setting problem (sections 3–4). A success here followed by a storage 403 during the scan is a network problem (section 5).

#7. Operations and monitoring

• Audit: vended credential requests and table access by the service principal show up in system.access.audit. Save a query filtered on its application ID and review it regularly.

SELECT event_time, action_name, request_params, response.status_code
FROM   system.access.audit
WHERE  service_name = 'unityCatalog'
  AND  user_identity.email = '<sp-application-id>'
  AND  event_date >= current_date() - INTERVAL 7 DAYS
ORDER  BY event_time DESC;

Field values for service principals vary, so check one known request to confirm how the principal appears.

• Ownership: a named team owns the exposed schema and approves new tables.
• Schema changes are communicated to consumers. Column renames are safe with column mapping, but type changes can still break readers.
• External engines don't run maintenance (snapshot expiry, orphan cleanup) on managed Iceberg tables. Databricks owns maintenance.
• Offboarding: revoking EXTERNAL USE SCHEMA stops new credentials immediately. Already-vended tokens stay valid until they expire (about an hour by default).

#Prevent it next time

• Manage EXTERNAL USE SCHEMA grants in Terraform alongside the other grants, so they're reviewed and can't be added ad hoc.
• Keep a single exposed schema per consumer, so revoking access is one statement.
• Check for row filters, column masks and the network path at design review, before anyone enables Iceberg reads on a table.