Home / Databricks on Azure

Databricks on AzureHow-toIntermediate

Optimization: when Photon pays off on Azure Databricks - test it per job, not per workspace

Photon charges more DBUs per hour but can finish SQL and DataFrame work much faster. Whether it saves money depends on the job. Check how much of a job Photon can run, compare cost per run with and without it, and keep it off for jobs dominated by Python UDFs, RDD code or tiny data.

Deepak Bhardwaj2 min read

Applies toAzure Databricks classic clusters with Photon option; Databricks SQL warehouses and serverless use Photon by default

How to think about it

  • Photon is a native vectorized engine for Spark SQL and DataFrame operations.
  • On classic compute, enabling Photon increases the DBU rate.
  • Saving = (run time without Photon × rate) − (run time with Photon × higher rate). Only real runs tell you.

Likely gains vs little gain

Likely to gain Little or no gain
Large scans, filters, aggregations, joins Python UDFs and pandas code on the driver
MERGE, UPDATE, DELETE on Delta RDD-based code
Writing Delta/Parquet Small data where startup dominates
Wide tables, many columns Jobs waiting on external systems (APIs, JDBC sources)

Check Photon coverage

In the Spark UI SQL tab (or query profile), operators run by Photon are marked (e.g. Photon... nodes). If a big part of the plan falls back to non-Photon operators, gains will be limited.

Common fallbacks: unsupported UDFs, certain functions or data types. Rewriting a UDF with built-in functions often lets Photon run the whole stage (see the UDF optimization post).

Run an A/B test

  1. Pick a job with stable input size.
  2. Run 3 times without Photon, 3 times with (same cluster size).
  3. Compare DBUs and VM time per run.
SELECT usage_metadata.job_run_id,
       product_features.is_photon,
       SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_metadata.job_id = '<job_id>'
  AND usage_date >= current_date() - 14
GROUP BY ALL
ORDER BY job_run_id DESC;

Also consider a smaller cluster with Photon: faster engines sometimes let you drop nodes.

Set policy by evidence

Result Action
Clearly cheaper per run Enable in the job cluster definition
Faster but more expensive Enable only if the deadline needs it
No real gain Leave off

Cluster policies can allow or require Photon per job category.

Tip

Re-test after Databricks Runtime upgrades. Photon coverage grows over time, so a job that didn't benefit last year might now.

Checklist

  • Job's Photon coverage checked in Spark UI
  • UDF fallbacks reviewed
  • A/B cost per run measured
  • Smaller cluster with Photon tried
  • Decision recorded per job; re-tested after upgrades

Related fixes