Databricks job fails with \"The output of the notebook is too large\": find the noisy cells and cut job output

#Symptom

A scheduled notebook job fails, usually after running for a long time. The run's error says something like:

Run result unavailable: job failed with error message
The output of the notebook is too large.

The same notebook runs fine when you run it by hand. Often the data has been written and the notebook only failed at the end. That makes things worse, because a retry can write the data again.

#Why it happens

Job runs keep a limit on the total output that a notebook can return: every print(), display(), show(), warning and library install message, added together. When the total passes the limit, the run is marked failed. The limit has been 20 MB for job clusters in older documentation and is higher in newer docs. Check the current notebook limits for your workspace instead of relying on a number.

Interactive runs don't fail the same way, because the notebook UI truncates large cell output. That's why the error shows up only in jobs.

Common causes:

#Step 1: confirm it's the output limit

Get the failed run and read its state message:

databricks jobs list-runs --job-id 123 --completed-only --limit 10 -p prod -o json \
  | jq -r '.[] | [.run_id, .state.result_state, .state.state_message] | @tsv'

Then look at the task run. For multi-task jobs, use the task's own run_id from tasks[]:

databricks jobs get-run 987654321 -p prod -o json \
  | jq -r '.tasks[] | [.task_key, .run_id, .state.result_state, .state.state_message] | @tsv'

databricks jobs get-run-output 987654322 -p prod -o json | jq '{error, error_trace, metadata: .metadata.state}'

If the message is about output size, keep going. If you see OutOfMemoryError or "driver is not responding", it's a different problem: driver memory, not output size.

#Step 2: find the cells that print the most

Open the failed run from Jobs & Pipelines > your job > Runs. The notebook snapshot shows output for each cell. Scroll and look for very long cell outputs. Usually one cell causes most of the output.

To search the code instead, export the notebook and look for the usual causes:

databricks workspace export /Workspace/Shared/etl/load_orders ./load_orders.py --format SOURCE -p prod
grep -nE 'print\(|display\(|\.show\(|%pip|awaitTermination|awaitAnyTermination|collect\(\)' ./load_orders.py

Check what the notebook calls too: %run notebooks and dbutils.notebook.run children add to the same run's output (for %run) or to their own run (for dbutils.notebook.run).

#Step 3: cut the output

#Remove or guard debug output

Keep debug output in the code, but turn it off for scheduled runs with a widget or job parameter:

dbutils.widgets.dropdown("debug", "false", ["true", "false"])
DEBUG = dbutils.widgets.get("debug") == "true"

df = spark.read.table("main.sales.orders_bronze")
if DEBUG:
    display(df.limit(20))

Never collect() a large DataFrame just to print it. If you need a sample, use df.limit(20).show(truncate=False).

#Quieter library installs

%pip install -q -r /Workspace/Shared/etl/requirements.txt

Better still, declare the libraries on the job task or the serverless job environment. Then nothing is installed from inside the notebook.

#Lower Python log levels

Many SDKs (Azure, HTTP clients) log a lot at INFO or DEBUG. Raise their level at the top of the notebook:

import logging

logging.basicConfig(level=logging.WARNING, force=True)
for name in ["azure", "urllib3", "py4j"]:
    logging.getLogger(name).setLevel(logging.WARNING)

#Record run details in a table, not in output

If people read the output to see row counts or progress, write that to a small Delta table. Nobody has to scroll through output, and you can query the history:

from pyspark.sql import functions as F

run_id = dbutils.widgets.get("run_id")  # map {{job.run_id}} to this task parameter

metrics = spark.createDataFrame(
    [(run_id, "orders_silver", rows_written)],
    "run_id string, step string, rows long",
).withColumn("logged_at", F.current_timestamp())

metrics.write.mode("append").saveAsTable("ops.monitoring.job_step_metrics")

#Streaming jobs

A streaming notebook that waits on queries can build up large progress output over a long run. For scheduled incremental loads, use trigger(availableNow=True). The query processes the backlog and stops, so there's no long-running wait. Watch progress with the job's streaming backlog metrics or a StreamingQueryListener that writes to a table. Don't print lastProgress in a loop.

#JAR tasks

For Scala or Java JAR tasks, stop the driver's stdout from being returned to the job. Add this to the job cluster's Spark config:

spark.databricks.driver.disableScalaOutput true

This applies to JAR output only. It doesn't silence Python print(). After you set it, logs from println are only in the driver logs (log delivery or the Spark UI), not in the run output.

Warning

If the notebook failed after its writes finished, a retry or a repair-run reruns the whole task. Check DESCRIBE HISTORY on the target tables before you rerun, and make the write idempotent (MERGE, or replaceWhere for a partition) so that a second run doesn't append the same rows twice.

#Step 4: rerun and verify

Run the fixed job once by hand with production-sized data, then check that the run succeeded and that the target table got one write for that run:

DESCRIBE HISTORY main.sales.orders_silver LIMIT 5;

The operation, operationMetrics.numOutputRows and timestamp columns show whether the rerun appended data a second time.

#Prevent it next time

• Treat display(), show() and print() in loops as review findings in pull requests for job notebooks.
• Add a debug parameter to shared notebooks, and keep it set to false in job definitions and bundles.
• Install libraries through job or serverless environments, not with %pip in production notebooks.
• Write row counts and step timings to a monitoring table instead of the notebook output.
• Test with a full-size data load before go-live. Small test datasets don't produce enough output to hit the limit.