Applies toAzure Databricks classic clusters and SQL warehouses (disk cache), PySpark/Scala DataFrame caching
Two caches compared
| Disk cache (formerly Delta cache) | Spark cache (cache() / persist()) | |
|---|---|---|
| What it stores | Copies of remote data files on local SSD | Computed DataFrame partitions in memory/disk |
| Enabled | Automatically on supported VM types; configurable | Only when code calls it |
| Freshness | Detects file changes automatically | Stale until uncached/recomputed |
| Memory pressure | Uses local disk | Uses executor memory, can cause spill/OOM |
| Best for | Repeated reads of the same tables | Reusing an expensive intermediate result several times in one job |
Disk cache
- Works best on VM types with fast local SSD (many storage-optimized and
d-suffixed sizes). - Check or set:
spark.conf.get("spark.databricks.io.cache.enabled"). - Pre-warm for BI-like workloads:
CACHE SELECT * FROM gold.sales WHERE order_date >= current_date() - 30; - SQL warehouses use it automatically.
Spark cache: use with care
Good use:
dim = spark.table("silver.customers").filter("active").select("id", "segment")
dim.cache()
dim.count() # materialise once
a = facts1.join(dim, "id")
b = facts2.join(dim, "id")
dim.unpersist() # free memory when done
Bad uses:
| Pattern | Problem |
|---|---|
cache() on a DataFrame used once | Extra work and memory for nothing |
| Caching huge tables | Evicts other data, causes spill/OOM |
Never calling unpersist() in long notebooks | Memory stays full |
| Caching then writing back to the same table | Stale data risk |
Find caching in the Spark UI
- Storage tab: cached DataFrames, size in memory/disk, fraction cached.
- If only a small fraction fits, the cache is mostly overhead.
Tip
In production jobs, search for
.cache()and.persist(). Remove those not reused at least twice, and addunpersist()for the rest. It often fixes memory errors as well as speeding jobs up.
Checklist
- VM types with local SSD where reads repeat
- Disk cache enabled and verified
- Spark cache only for multi-use intermediates
-
unpersist()after use - Storage tab checked for oversized caches