Home / Data Platform / Databricks & Spark

Data PlatformCheat sheetBeginner

spark-submit cheat sheet: memory, cores, deploy mode and the errors you see first

A practical spark-submit reference for beginners - what each option does, ready-to-use commands for local, YARN and Kubernetes, simple sizing math, how to find the logs and what the first errors mean.

Deepak Bhardwaj5 min read

Applies toApache Spark 3.x and later, YARN, Kubernetes, PySpark

What this is for

spark-submit starts a Spark application on a cluster. You'll use it, or see it in a scheduler's logs, on most self-managed Spark platforms. This page covers the options you'll use on most days. Managed platforms such as Databricks set many of these for you.

The basic shape

spark-submit [options] app.py [arguments for your app]

Options go before the script. Anything after the script goes to your app, not to Spark. That's a common reason a setting appears to be ignored.

Options you'll use most

Option Meaning Example
--master Where to run local[4], yarn, k8s://https://host:6443
--deploy-mode Where the driver runs client or cluster
--name Name shown in the UI --name daily-orders
--driver-memory Memory for the driver 4g
--executor-memory Heap per executor 8g
--executor-cores Cores per executor 4
--num-executors Executors (YARN) 10
--conf Any Spark setting --conf spark.sql.shuffle.partitions=200
--py-files Extra Python code --py-files libs.zip
--jars Extra JAR files --jars postgresql.jar
--packages JARs by Maven coordinate --packages org.apache.kafka:kafka-clients:3.7.0
--files Files sent to each executor --files app.conf
--queue YARN queue --queue etl

Ready-to-use commands

Run on your own machine, using 4 cores:

spark-submit --master "local[4]" --driver-memory 4g app.py --date 2026-11-06

Run on YARN, for production:

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --name daily-orders \
  --queue etl \
  --driver-memory 4g \
  --executor-memory 8g \
  --executor-cores 4 \
  --num-executors 10 \
  --conf spark.sql.shuffle.partitions=200 \
  --py-files libs.zip \
  app.py --date 2026-11-06

Run on Kubernetes:

spark-submit \
  --master k8s://https://k8s-api.contoso.com:6443 \
  --deploy-mode cluster \
  --name daily-orders \
  --conf spark.kubernetes.namespace=spark \
  --conf spark.kubernetes.container.image=registry.contoso.com/spark-app:1.4 \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --conf spark.executor.instances=5 \
  local:///opt/app/app.py

local:// means the file is already inside the image.

Client mode or cluster mode?

client cluster
Driver runs on The machine where you ran the command A node in the cluster
Logs appear In your terminal In the cluster's logs
If your session ends The job dies The job keeps running
Good for Development and debugging Scheduled production jobs

Simple sizing math

Start from the job's needs, then adjust using the Spark UI:

  1. Use 4-5 cores per executor. More often gives worse results.
  2. Leave some memory on each node for the OS and services.
  3. Remember the overhead: each executor uses its heap plus an overhead that's about 10% by default.

Example, with worker nodes of 16 cores and 64 GB:

Usable on a node:  15 cores, about 58 GB
Executors a node:  15 / 5 = 3
Per executor:      58 / 3 = about 19 GB total
Heap:              19 GB minus about 10% overhead = about 17 GB

So you'd set --executor-cores 5 --executor-memory 17g. Treat this as a starting point, not a final answer.

Let Spark scale the executor count when your cluster supports it:

--conf spark.dynamicAllocation.enabled=true \
--conf spark.dynamicAllocation.minExecutors=2 \
--conf spark.dynamicAllocation.maxExecutors=20 \
--conf spark.dynamicAllocation.shuffleTracking.enabled=true

Find the logs

# YARN: list applications, then fetch the logs
yarn application -list -appStates RUNNING,FAILED
yarn logs -applicationId application_1730000000000_0042 | less

# Stop a running app
yarn application -kill application_1730000000000_0042

# Kubernetes
kubectl get pods -n spark -l spark-role=driver
kubectl logs -n spark <driver-pod> --tail=200

For how to read the UI once a job runs, see the Spark UI guide.

First errors and what they mean

Error Usual cause Fix
ClassNotFoundException A JAR is missing Add it with --jars or --packages
ModuleNotFoundError (Python) Module isn't on the executors Send it with --py-files, or install it in the image
Container killed by YARN for exceeding memory limits Off-heap use exceeded overhead Raise spark.executor.memoryOverhead
java.lang.OutOfMemoryError: Java heap space Heap too small, or data skew See the OOM guide
App stays ACCEPTED The YARN queue has no free capacity Try another queue or wait; check the queue's limits
Python in worker has different version Driver and workers use different Pythons Set PYSPARK_PYTHON to the same path
Initial job has not accepted any resources You asked for more than a node can supply Lower executor memory or cores
Settings have no effect Options were placed after the script Move them before it

Warning

Don't put passwords on the command line. They show up in process lists and the Spark UI's Environment tab. Read them from a secret store inside your app.

Check what Spark really used

The Environment tab in the Spark UI lists every setting the app ran with. Print them to the log too:

spark-submit --verbose --master "local[2]" app.py

Prevent it next time

  • Keep a working command for each job in Git, not in someone's shell history.
  • Set --name to something meaningful, so you can find the app later.
  • Change one setting at a time, and record the run time and result.
  • Next read: shuffle partitions and AQE.

Related fixes