Azure Databricks logs cheat sheet: where to find driver, executor, init script, event and audit logs

A job failed, the cluster has already terminated, and someone asks "what does the log say?". This page maps each question to the log that answers it and shows how to keep those logs after the compute is gone.

#Which log answers which question

Warning

Driver and executor logs in the UI are only available while the compute exists. Job clusters are removed shortly after the run, so their logs are gone unless you configured compute log delivery. Turn it on before you need it.

#Driver and executor logs

#In the UI

• Driver logs tab on the cluster page shows three streams:
  • stdout: print() output and anything written to standard out.
  • stderr: Python tracebacks, warnings, and most Python logging output by default.
  • log4j: Spark and JVM messages. Search here for OutOfMemoryError, FetchFailedException and ExecutorLostFailure.
• Spark UI > Executors lists each executor with stdout and stderr links. Use them when a task fails on one node but the driver only shows Task failed ... lost.
• For a job run, open the run, pick the task, and use Compute > View details / Logs (labels vary by release).

#Make your own messages easy to find

Write to the Python logger with a clear prefix so you can grep the delivered logs later:

import logging

log = logging.getLogger("etl.orders")
log.setLevel(logging.INFO)
if not log.handlers:
    h = logging.StreamHandler()
    h.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s"))
    log.addHandler(h)

log.info("loaded %s rows into silver.orders", df.count())

Tip

Don't log every row or call df.show() inside loops. Large driver logs slow the driver and fill its disk.

#Compute log delivery: keep logs after termination

Compute log delivery copies driver, executor, event and init script logs to a path you choose. Logs are delivered every five minutes and archived hourly. Use a Unity Catalog volume; DBFS is the legacy option.

Requirements for a volume destination:

• Unity Catalog compute in Standard access mode, or Dedicated access mode assigned to a user. Dedicated assigned to a group is not supported.
• The compute owner (or assigned user) needs READ VOLUME and WRITE VOLUME on the volume.

CREATE VOLUME IF NOT EXISTS ops.logs.cluster_logs;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME ops.logs.cluster_logs TO `sp-etl-prod`;

Add it to the cluster spec, job cluster or Asset Bundle:

"cluster_log_conf": {
  "volumes": { "destination": "/Volumes/ops/logs/cluster_logs" }
}

In the UI it is under Advanced > Logging. To give every job cluster the same setting, fix it in your cluster policy with the cluster_log_conf.* attributes. Check the policy reference for the destination types your workspace accepts, because volume support in policies arrived later than DBFS.

#What lands in the destination

Databricks creates a folder named after the cluster_id. Under it you will usually see:

/Volumes/ops/logs/cluster_logs/<cluster-id>/
  driver/        stdout, stderr, log4j-active.log, rotated log4j-*.log.gz
  executor/      per-executor stdout/stderr
  eventlog/      Spark event logs (feed for a Spark history replay)
  init_scripts/  <cluster-id>_<container-ip>/<timestamp>_<id>_<script>.sh.stdout.log / .stderr.log

Folder names can vary by runtime; list the folder to confirm on your build.

#Read delivered logs

# Find the cluster ID of a failed job run (multi-task jobs: one per task)
databricks jobs get-run 123456789 -p prod | jq '.tasks[] | {task_key, cluster: .cluster_instance.cluster_id, state: .state.result_state}'

# List and download logs
databricks fs ls dbfs:/Volumes/ops/logs/cluster_logs/0612-081122-abcd1234/driver -p prod
databricks fs cp dbfs:/Volumes/ops/logs/cluster_logs/0612-081122-abcd1234/driver/stderr ./stderr.txt -p prod

Or search from a notebook:

path = "/Volumes/ops/logs/cluster_logs/0612-081122-abcd1234/driver/log4j-active.log"
(spark.read.text(path)
      .filter("value rlike 'ERROR|OutOfMemory|Exception'")
      .show(50, truncate=False))

Caution

Anyone who can create a cluster with log delivery can read stdout/stderr from global init scripts. Never echo secrets in init scripts, and restrict who can read the log volume.

#Cluster event log

The event log records starts, terminations with a reason code, resizes, spot losses, init script start/finish and driver health.

databricks clusters events 0612-081122-abcd1234 -p prod \
  | jq -r '.[] | [.timestamp, .type, (.details.reason.code // "")] | @tsv' | head -50

The JSON output shape varies by CLI version. If jq returns nothing, check the raw output first. Look for:

#Job run output

# Task-level output and error (use the task's run_id, not the parent job run_id)
databricks jobs get-run-output 987654321 -p prod | jq '{error, error_trace, notebook_output}'

History after the cluster is gone, from system tables:

SELECT job_id, run_id, task_key, result_state, termination_code, period_start_time, period_end_time
FROM system.lakeflow.job_task_run_timeline
WHERE period_start_time > now() - INTERVAL 1 DAY
  AND result_state NOT IN ('SUCCEEDED')
ORDER BY period_start_time DESC;

Column names in system tables change between releases; run DESCRIBE on the table if a column is missing.

#Audit logs: who did what

SELECT event_time, user_identity.email, service_name, action_name, request_params, response.status_code
FROM system.access.audit
WHERE event_date >= current_date() - 2
  AND service_name IN ('clusters', 'jobs', 'unityCatalog')
  AND action_name IN ('delete', 'edit', 'permanentDelete', 'updatePermissions', 'deleteTable')
ORDER BY event_time DESC
LIMIT 200;

To send the same events to Log Analytics, create a diagnostic setting on the workspace (Premium tier):

WS_ID=$(az databricks workspace show -g rg-dbx-prod -n dbx-prod --query id -o tsv)
LAW_ID=$(az monitor log-analytics workspace show -g rg-ops -n law-ops --query id -o tsv)

az monitor diagnostic-settings create --name dbx-to-law \
  --resource "$WS_ID" --workspace "$LAW_ID" \
  --logs '[{"category":"clusters","enabled":true},{"category":"jobs","enabled":true},{"category":"unityCatalog","enabled":true}]'

Then query in KQL:

DatabricksJobs
| where TimeGenerated > ago(1d)
| where ActionName in ("delete", "reset", "cancel")
| project TimeGenerated, Identity, ActionName, RequestParams

Run az monitor diagnostic-settings categories list --resource "$WS_ID" to see the categories your workspace supports.

#Serverless compute

Serverless notebooks and jobs have no cluster page, no driver log tab and no compute log delivery. Use:

• The job run page and get-run-output for errors.
• Query history and the query profile for SQL and DataFrame performance.
• system.access.audit and system.lakeflow.* tables for history.

#Prevent it next time

• Enforce cluster_log_conf to a Unity Catalog volume in every job cluster policy.
• Grant read on the log volume to the on-call group, not everyone.
• Add a lifecycle rule or scheduled cleanup on the log volume's storage path so it doesn't grow forever.
• Enable the workspace diagnostic setting or confirm system.access is enabled before an incident, not during one.
• Log with a consistent prefix (etl.<pipeline>) so delivered logs are searchable.