Azure Databricks first steps: workspace tour, choosing compute and running your first notebook

A new Databricks workspace has a lot of buttons. You only need a few of them on day one. This post walks through them in the order you will use them.

#The sidebar in one table

#Pick the right compute

Tip

If serverless notebooks are enabled in your workspace, start there. You avoid most cluster configuration questions in week one.

#Create a small all-purpose cluster

Compute > Create compute:

• Policy: Personal Compute (if your admins provide it) or the policy you were given.
• Access mode: Standard (shared) or Dedicated (single user); both work with Unity Catalog.
• Runtime: the latest LTS version.
• Node type: a small general-purpose VM; single node is enough for learning.
• Terminate after 30 minutes of inactivity.

Or with the CLI:

databricks clusters create --json '{
  "cluster_name": "alice-learning",
  "spark_version": "15.4.x-scala2.12",
  "node_type_id": "Standard_D4ds_v5",
  "num_workers": 0,
  "spark_conf": {"spark.master": "local[*]", "spark.databricks.cluster.profile": "singleNode"},
  "custom_tags": {"ResourceClass": "SingleNode", "owner": "alice"},
  "autotermination_minutes": 30,
  "data_security_mode": "SINGLE_USER"
}'

Runtime and VM names change; pick valid ones with databricks clusters spark-versions and databricks clusters list-node-types.

#Your first notebook

Workspace > Create > Notebook, attach it to serverless or your cluster.

# 1. Read a sample dataset
df = spark.read.table("samples.nyctaxi.trips")
display(df.limit(10))

# 2. A simple aggregation
from pyspark.sql import functions as F
daily = (df.groupBy(F.to_date("tpep_pickup_datetime").alias("day"))
           .agg(F.count("*").alias("trips"), F.round(F.avg("fare_amount"), 2).alias("avg_fare"))
           .orderBy("day"))
display(daily)

The samples catalog is available in most Unity Catalog workspaces. If it is missing, upload a CSV through Data ingestion instead.

Write the result to your own schema (ask your admin which catalog you can use):

spark.sql("CREATE SCHEMA IF NOT EXISTS sandbox.alice")
daily.write.mode("overwrite").saveAsTable("sandbox.alice.taxi_daily")

Then query it in SQL, in a %sql cell:

%sql
SELECT * FROM sandbox.alice.taxi_daily ORDER BY trips DESC LIMIT 5;

#Names in Unity Catalog

Tables are addressed as catalog.schema.table. Set defaults to save typing:

USE CATALOG sandbox;
USE SCHEMA alice;

#Work in Git from the start

Workspace > Create > Git folder, paste your repository URL. Notebooks you commit there can be reviewed and deployed. Personal notebooks outside Git folders are easy to lose track of.

#Useful CLI commands

databricks auth login --host https://adb-1234567890123456.7.azuredatabricks.net
databricks current-user me
databricks clusters list
databricks workspace list /Users/alice@contoso.com

#Beginner mistakes

#Prevent it next time

• Ask for a personal sandbox schema and a compute policy from your platform team on day one.
• Keep notebooks in Git folders.
• Check Compute before you log off for anything you started that is still running.