Databricks notebook basics: magic commands, widgets, dbutils and running one notebook from another

#Language magics

A notebook has a default language; each cell can switch with a magic on its first line.

%sql
SELECT current_catalog(), current_schema(), current_user();

The result of the last SQL cell is available in Python as _sqldf.

#Install libraries

%pip install great-expectations==1.2.0
dbutils.library.restartPython()

Put %pip cells at the top. Restarting Python clears variables. For jobs, prefer libraries declared on the job/environment so every run is identical.

#Widgets (parameters)

dbutils.widgets.text("run_date", "2026-10-28", "Run date")
dbutils.widgets.dropdown("env", "dev", ["dev", "test", "prod"], "Environment")

run_date = dbutils.widgets.get("run_date")
env = dbutils.widgets.get("env")

In SQL cells use named parameter markers, for example :run_date. Job parameters with the same name override widget defaults, which makes widgets the standard way to parameterise notebook tasks.

dbutils.widgets.removeAll()   # clean up while developing

#Files with dbutils.fs

dbutils.fs.ls("/Volumes/sales_dev/orders/landing/")
dbutils.fs.head("/Volumes/sales_dev/orders/landing/sample.csv", 500)
dbutils.fs.cp("/Volumes/sales_dev/orders/landing/a.csv", "/Volumes/sales_dev/orders/archive/a.csv")
dbutils.fs.mkdirs("/Volumes/sales_dev/orders/archive/2026-10-29")
dbutils.fs.rm("/Volumes/sales_dev/orders/tmp/", recurse=True)

Caution

dbutils.fs.rm(..., recurse=True) deletes immediately with no recycle bin. Print the ls first, and never point it at a variable you have not checked.

Volumes paths also work with plain Python file APIs:

with open("/Volumes/sales_dev/orders/landing/notes.txt") as f:
    print(f.read()[:200])

#Secrets

pwd = dbutils.secrets.get(scope="kv-sales", key="sql-password")
dbutils.secrets.listScopes()
dbutils.secrets.list("kv-sales")

Secret values are redacted ([REDACTED]) if printed. Never copy them into a widget or a table.

#Running other notebooks

Child notebook:

import json
dbutils.widgets.text("run_date", "")
# ... work ...
dbutils.notebook.exit(json.dumps({"rows": 1234}))

Parent:

import json
result = json.loads(dbutils.notebook.run("./child", 600, {"run_date": "2026-10-28"}))
print(result["rows"])

For anything more than a couple of steps, use job tasks instead: each task gets its own retries, logs and repair.

#Handy notebook features

#Notebook context

ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
print(spark.conf.get("spark.databricks.clusterUsageTags.clusterName", "serverless"))
print(spark.sql("select current_user()").first()[0])

#Prevent it next time

• Parameters through widgets, defaults safe for dev.
• Shared code in Python files in a Git folder (importable modules), not long %run chains.
• Secrets only through dbutils.secrets.