Spark on Kubernetes production checklist: memory overhead, local storage, dynamic allocation and executor decommissioning

Spark on Kubernetes fails differently from YARN. Pods are OOMKilled because the container limit is lower than Spark thinks it is. Shuffle data lives on executors that the autoscaler removes. Logs vanish with the pod. Use this checklist before a Spark workload goes to production on a shared cluster. Most items are --conf settings, so they work with spark-submit and with a SparkApplication spec.

#Identity and images

• Dedicated namespace and service account for drivers. The driver creates executor pods, services and config maps, so its service account needs those verbs, and only in its own namespace.

kubectl create namespace spark-jobs
kubectl -n spark-jobs create serviceaccount spark-driver
kubectl -n spark-jobs create rolebinding spark-driver-edit \
  --clusterrole=edit --serviceaccount=spark-jobs:spark-driver

--conf spark.kubernetes.namespace=spark-jobs
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-driver

• Pinned image tags, never latest. Set spark.kubernetes.container.image to an immutable tag or digest, and set spark.kubernetes.container.image.pullPolicy=IfNotPresent. Large images pulled on every executor start slow down scale-out.
• Cloud storage access through workload identity (IRSA or EKS Pod Identity, Entra Workload ID, GKE Workload Identity), not access keys in Spark conf. Spark conf is visible in the UI and event logs.

#Pod memory and CPU

• Size the container, not just the heap. Executor pod memory = spark.executor.memory + overhead + spark.executor.pyspark.memory (if set). Overhead is spark.executor.memoryOverhead, or spark.executor.memoryOverheadFactor times the heap. The default factor is 0.10 for JVM jobs, and Spark on Kubernetes uses 0.40 for non-JVM (PySpark, R) jobs. Older releases used spark.kubernetes.memoryOverheadFactor; check your version's docs.
• Know what lives in overhead: Netty buffers, off-heap (spark.memory.offHeap.size is added separately when enabled), Python workers, native libraries, and tmpfs local dirs if you use them. If pods are OOMKilled while the Spark UI shows heap to spare, overhead is too small.

kubectl -n spark-jobs get pod <executor-pod> \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# OOMKilled = the container limit was hit (fix overhead), not a JVM OutOfMemoryError

• CPU requests and limits are deliberate. spark.kubernetes.executor.request.cores can be fractional, so it packs pods tighter than spark.executor.cores, which sets task slots. Apply a CPU limit with spark.kubernetes.executor.limit.cores only if your platform requires one. CFS throttling on a tight limit shows up as slow GC and long task times.
• Driver sized for the plan, not the data. Broadcasts, collect() and large file listings land on the driver. Give it its own overhead with spark.driver.memoryOverhead.

#Local storage for shuffle and spill

• Shuffle and spill go to real disks. By default, Spark local dirs are emptyDir volumes on the node's root disk, where they compete with images and logs. That can lead to evictions for ephemeral-storage pressure. Mount NVMe or a dedicated volume instead. Volumes named with the spark-local-dir- prefix are used as local dirs:

--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName=OnDemand
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.storageClass=gp3
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit=200Gi
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path=/data/spark-local
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.readOnly=false

• spark.kubernetes.local.dirs.tmpfs=true only when you have accounted for the RAM. It makes shuffle memory-backed, and that usage counts against the pod's memory limit.
• PVC reuse for churny jobs. spark.kubernetes.driver.ownPersistentVolumeClaim=true and spark.kubernetes.driver.reusePersistentVolumeClaim=true (3.2+) let replacement executors reattach existing on-demand claims instead of provisioning new volumes.

Warning

Ephemeral-storage evictions kill executors in the middle of a shuffle and look like FetchFailedException storms. Set an ephemeral-storage request on executor pods through a pod template, so the scheduler places them on nodes with room.

#Dynamic allocation without an external shuffle service

• Shuffle tracking on. Kubernetes has no node-level external shuffle service by default, so dynamic allocation depends on tracking which executors hold shuffle data:

--conf spark.dynamicAllocation.enabled=true
--conf spark.dynamicAllocation.shuffleTracking.enabled=true
--conf spark.dynamicAllocation.shuffleTracking.timeout=30min
--conf spark.dynamicAllocation.minExecutors=2
--conf spark.dynamicAllocation.maxExecutors=50
--conf spark.dynamicAllocation.executorIdleTimeout=120s

• A shuffle-tracking timeout is set. Without one, an executor that holds shuffle files for a cached or long-lived DataFrame is never released, so a job that has been idle for a while still holds its peak executor count.
• Executor pod creation is throttled for big jobs. spark.kubernetes.allocation.batch.size (default 10) and spark.kubernetes.allocation.batch.delay control how fast pods are requested. Raising them speeds up scale-out, but it can flood the API server and the cluster autoscaler.
• Stateful streaming queries keep a fixed executor count. Losing executors forces state stores to reload from the checkpoint.

#Spot nodes and graceful decommissioning

• Decommissioning migrates shuffle and cached blocks before the pod dies:

--conf spark.decommission.enabled=true
--conf spark.storage.decommission.enabled=true
--conf spark.storage.decommission.shuffleBlocks.enabled=true
--conf spark.storage.decommission.rddBlocks.enabled=true
--conf spark.storage.decommission.fallbackStorage.path=s3a://spark-fallback/shuffle/

• The pod's grace period covers the migration. Set terminationGracePeriodSeconds in the executor pod template. The default of 30 seconds is rarely enough to move gigabytes of shuffle data. The node termination handler or Karpenter must drain the node, rather than hard-stop it, so the pods actually receive SIGTERM.
• Drivers never run on spot. Losing the driver loses the whole application. Use spark.kubernetes.driver.node.selector.<label> (3.3+), or a driver pod template with node affinity to on-demand nodes, and allow executors onto spot with tolerations.
• The fallback storage path has a lifecycle rule. It is not cleaned up if the application dies.

#Pod templates and scheduling

• Pod templates for what Spark conf can't express: tolerations, affinity, ephemeral-storage requests, sidecars and priorityClassName. Set them with spark.kubernetes.driver.podTemplateFile and spark.kubernetes.executor.podTemplateFile. Spark overrides some fields, such as the memory request and the container image.
• Gang scheduling on shared clusters. Without it, ten jobs that each have their driver running can starve one another of executors. Use Volcano or Apache YuniKorn with spark.kubernetes.scheduler.name, and set queues and quotas per team.
• A ResourceQuota per namespace, so one runaway maxExecutors can't take the whole cluster.

#Logs, history and cleanup

• Event logs go to object storage (spark.eventLog.enabled=true, with spark.eventLog.dir set to an s3a://, abfss:// or gs:// path), with a Spark History Server reading them. The driver UI disappears when the driver pod completes.
• Container logs are shipped by a node-level agent such as Fluent Bit. Executor pods are deleted on completion by default (spark.kubernetes.executor.deleteOnTermination=true). Set it to false only temporarily, while you debug.
• Completed driver pods are cleaned up. They stay in Completed state until something removes them. The Spark Operator has TTL settings; with plain spark-submit, run a scheduled cleanup:

kubectl -n spark-jobs delete pod --field-selector=status.phase==Succeeded -l spark-role=driver

#Quick failure map

Setting names and defaults change between Spark minor versions. Check the "Running on Kubernetes" page for the version in your image before you copy a conf block.