Azure Databricks in Log Analytics: KQL cheat sheet for job failures, cluster changes, logins and secret access

#When to use this

Use these queries when your Databricks audit events already go to Log Analytics and you need to answer "what failed?" or "who changed this?". They're also useful for joining Databricks events with other Azure logs or building Azure Monitor alerts.

Databricks recommends the system.access.audit system table for audit logs in general. Use KQL when your alerting, Sentinel rules or on-call habits live in Azure Monitor.

Keep these limits in mind:

• Diagnostic logs require the Premium plan and are configured per workspace.
• Events typically appear within 15 minutes, with no guaranteed delivery time. Don't use them as your only fast paging signal.
• Account-level events aren't included. Use system.access.audit for those.
• RequestParams is truncated above 100 KB, so parse_json can return nothing useful on very large requests.

#Turn it on

WS_ID=$(az databricks workspace show -g rg-dbx-prod -n dbx-prod --query id -o tsv)
LAW_ID=$(az monitor log-analytics workspace show -g rg-ops -n law-ops --query id -o tsv)

# See which categories this workspace supports
az monitor diagnostic-settings categories list --resource "$WS_ID" -o table

az monitor diagnostic-settings create --name dbx-to-law --resource "$WS_ID" --workspace "$LAW_ID" \
  --logs '[{"category":"jobs","enabled":true},{"category":"clusters","enabled":true},
           {"category":"accounts","enabled":true},{"category":"secrets","enabled":true},
           {"category":"workspace","enabled":true},{"category":"unityCatalog","enabled":true}]'

#Tables and columns

All of these share TimeGenerated, ActionName, Identity, RequestParams, Response, SourceIPAddress, UserAgent and _ResourceId. The last one tells you which workspace the event came from.

Identity, RequestParams and Response are JSON held in string columns, so parse them first:

DatabricksJobs
| where TimeGenerated > ago(1h)
| extend Who = tostring(parse_json(Identity).email),
         Params = parse_json(RequestParams),
         Status = toint(parse_json(Response).statusCode),
         Error = tostring(parse_json(Response).errorMessage)
| take 20

Tip

Automated actions, such as autoscaling resizes and scheduled job launches, are logged under the user System-User. Add | where Who != "System-User" when you only want actions taken by people or service principals.

#Is data arriving?

Run this first. An empty result doesn't prove nothing happened if the category was never enabled.

union withsource=SourceTable Databricks*
| where TimeGenerated > ago(24h)
| summarize Events = count(), LastEvent = max(TimeGenerated) by SourceTable, _ResourceId
| order by SourceTable asc

#Job runs

#Failed and canceled runs

runFailed covers runs that fail or are canceled. Check jobTerminalState to tell them apart.

DatabricksJobs
| where TimeGenerated > ago(24h)
| where ActionName == "runFailed"
| extend P = parse_json(RequestParams)
| project TimeGenerated,
          JobId = tostring(P.jobId),
          TaskKey = tostring(P.taskKey),
          TerminalState = tostring(P.jobTerminalState),
          RunId = tostring(P.runId),
          ParentRunId = tostring(P.multitaskParentRunId),
          ClusterId = tostring(P.clusterId),
          _ResourceId
| order by TimeGenerated desc

#Failure rate per job

DatabricksJobs
| where TimeGenerated > ago(7d)
| where ActionName in ("runSucceeded", "runFailed")
| extend JobId = tostring(parse_json(RequestParams).jobId)
| summarize Failed = countif(ActionName == "runFailed"),
            Succeeded = countif(ActionName == "runSucceeded") by JobId
| extend FailurePct = round(100.0 * Failed / (Failed + Succeeded), 1)
| where Failed > 0
| order by Failed desc

Events carry a taskKey, so on multi-task jobs read this as task-run failures, not whole-job failures.

#Who changed, deleted or canceled a job

DatabricksJobs
| where TimeGenerated > ago(7d)
| where ActionName in ("create", "update", "reset", "delete", "cancel", "cancelAllRuns",
                       "runNow", "repairRun", "changeJobAcl", "resetJobAcl")
| extend Who = tostring(parse_json(Identity).email)
| project TimeGenerated, Who, ActionName, SourceIPAddress, RequestParams
| order by TimeGenerated desc

#Clusters

#Who changed or removed a cluster

In these logs, delete means the cluster was terminated. permanentDelete means it was removed from the UI.

DatabricksClusters
| where TimeGenerated > ago(7d)
| where ActionName in ("edit", "delete", "permanentDelete", "restart", "changeOwner", "changeClusterAcl")
| extend P = parse_json(RequestParams), Who = tostring(parse_json(Identity).email)
| project TimeGenerated, Who, ActionName,
          ClusterId = tostring(P.cluster_id),
          TerminationReason = tostring(P.termination_reason),
          SourceIPAddress
| order by TimeGenerated desc

These events show API requests. For why a cluster failed to launch or terminated, use databricks clusters events <cluster-id>.

#Logins and tokens

#Failed logins by user and source IP

DatabricksAccounts
| where TimeGenerated > ago(24h)
| where ActionName in ("login", "aadBrowserLogin", "aadTokenLogin", "tokenLogin",
                       "oidcBrowserLogin", "samlLogin", "jwtLogin")
| extend Status = toint(parse_json(Response).statusCode),
         User = tostring(parse_json(RequestParams).user)
| where Status >= 400
| summarize Failures = count(), LastSeen = max(TimeGenerated) by User, SourceIPAddress, ActionName, Status
| order by Failures desc

#Personal access token activity

DatabricksAccounts
| where TimeGenerated > ago(30d)
| where ActionName in ("generateDbToken", "revokeDbToken", "updateDbToken", "changeDbTokenState")
| extend Who = tostring(parse_json(Identity).email)
| project TimeGenerated, Who, ActionName, SourceIPAddress, RequestParams
| order by TimeGenerated desc

#Secrets

#Secret reads per principal, scope and key

DatabricksSecrets
| where TimeGenerated > ago(7d)
| where ActionName == "getSecret"
| extend P = parse_json(RequestParams),
         Who = tostring(parse_json(Identity).email),
         Status = toint(parse_json(Response).statusCode)
| summarize Reads = count(), Denied = countif(Status == 403), LastRead = max(TimeGenerated)
    by Who, Scope = tostring(P.scope), Key = tostring(P.key)
| order by Reads desc

For scope and ACL changes, use the same pattern with createScope, deleteScope, putAcl, deleteAcl and deleteSecret.

#Workspace settings and Unity Catalog

#Who changed a workspace setting

DatabricksWorkspace
| where TimeGenerated > ago(30d)
| where ActionName in ("workspaceConfEdit", "setSetting", "deleteSetting")
| extend P = parse_json(RequestParams), Who = tostring(parse_json(Identity).email)
| project TimeGenerated, Who, ActionName,
          ConfKeys = tostring(P.workspaceConfKeys),
          ConfValues = tostring(P.workspaceConfValues),
          Setting = tostring(P.settingKeyName),
          SourceIPAddress
| order by TimeGenerated desc

#Grants and table deletes

DatabricksUnityCatalog
| where TimeGenerated > ago(7d)
| where ActionName in ("updatePermissions", "deleteTable")
| extend P = parse_json(RequestParams), Who = tostring(parse_json(Identity).email)
| project TimeGenerated, Who, ActionName,
          Object = coalesce(tostring(P.securable_full_name), tostring(P.full_name_arg)),
          Changes = tostring(P.changes)
| order by TimeGenerated desc

#Unity Catalog permission denials

DatabricksUnityCatalog
| where TimeGenerated > ago(24h)
| extend Status = toint(parse_json(Response).statusCode),
         Who = tostring(parse_json(Identity).email)
| where Status == 403
| summarize Denials = count(), LastSeen = max(TimeGenerated) by Who, ActionName
| order by Denials desc

#Alert on job failures

az extension add --name scheduled-query

az monitor scheduled-query create --name dbx-job-failures --resource-group rg-ops \
  --scopes "$LAW_ID" \
  --condition "count 'Failures' > 0" \
  --condition-query Failures="DatabricksJobs | where ActionName == 'runFailed'" \
  --evaluation-frequency 15m --window-size 30m --severity 2 \
  --action-groups "<action-group-resource-id>"

The window is wider than the frequency to allow for ingestion delay, so one failure can fire in more than one evaluation. This rule also fires on canceled runs. To exclude them, check the jobTerminalState values in your own data, then add a filter on the value they use.

#Common problems

#Prevent it next time

• Enable the diagnostic setting in your workspace deployment template, so new workspaces aren't missed.
• Save these queries in a shared query pack or workbook before an incident.
• Set table retention to match your audit policy, and check current Log Analytics pricing and limits before adding high-volume categories.
• Keep Databricks job notifications as the first alert, and use these queries to investigate and report.