Databricks Runtime 18 query results that changed in 2026 - NATURAL JOIN case matching, LATERAL EXPLODE nulls and windowed min/max on binary columns

#Why this matters

Errors get noticed. Changed results don't. When a maintenance update or runtime upgrade changes what a query returns, downstream tables quietly change. These three are documented in the Runtime 18 release notes and maintenance updates.

#1. NATURAL JOIN respects case-insensitivity

From the DBR 18 LTS notes (May/June 2026):

NATURAL JOIN now respects the spark.sql.caseSensitive configuration (default: false).

With the default setting, columns that differ only in case (ID and id) are now treated as the same column and used as a join key. Before, they weren't matched.

Effect: a NATURAL JOIN between tables with ID and id now joins on that column as well, so it can return fewer rows and different column sets.

Find candidates:

grep -rniE "natural\s+(left\s+|right\s+|full\s+)?(outer\s+)?join" notebooks/ sql/ jobs/

Restore the old behaviour temporarily with:

spark.sql.legacy.naturalJoinCaseSensitiveColumnMatching true

Better fix: replace NATURAL JOIN with an explicit JOIN ... USING (...) or ON clause. NATURAL JOIN silently changes meaning whenever someone adds a column.

#2. LEFT JOIN LATERAL with EXPLODE keeps outer rows

DBR 18 LTS fixed a bug where LEFT OUTER JOIN LATERAL dropped rows, and a June 2026 entry fixed null handling in those expressions when they contain EXPLODE.

Effect: queries of this shape can now return more rows (outer rows with nulls, as LEFT JOIN semantics require):

SELECT o.order_id, i.item
FROM orders o
LEFT JOIN LATERAL (SELECT explode(o.items) AS item) i;

If a downstream count or aggregate depended on the old (wrong) behaviour, it changes. The temporary revert for the row-dropping fix is:

spark.databricks.sql.optimizer.lateralJoinPreserveOuterSemantic true

Use it only to buy time; the new result is the correct one.

#3. Windowed min/max on BinaryType

The maintenance update of 24 July 2026 (18 LTS, 18.2 and 17.3 LTS) fixed windowed min/max on BinaryType columns, which:

returned the running global min/max instead of the per-frame value.

Effect: queries like this now return different (correct) values:

SELECT id, max(hash_bytes) OVER (PARTITION BY account ORDER BY ts ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)
FROM events;

Binary columns are often hashes, UUIDs stored as bytes, or encoded keys. If you used windowed min/max over them (for example for "latest key in a frame"), earlier results may have been wrong. Consider whether tables built before the fix need rebuilding.

#How to check your estate

1. Search code for NATURAL JOIN, JOIN LATERAL and windowed min(/max( over binary columns.
2. Compare results: run the critical queries on the old and new runtime (two job clusters with pinned runtime versions) against the same table version (VERSION AS OF) and diff the outputs.
3. Decide per query: accept the new result (and maybe backfill), or rewrite the query to state the intended behaviour explicitly.
4. Pin runtimes for critical jobs to a specific LTS version and move them on purpose, with a comparison run, rather than with every cluster restart.

#Checklist

• NATURAL JOINs found and replaced with explicit join keys
• LEFT JOIN LATERAL ... EXPLODE queries reviewed for row-count changes
• Windowed min/max over binary columns found; affected tables assessed for rebuild
• Old vs new runtime comparison run for critical pipelines
• Legacy flags, if used, removed with a date

#Sources

• Databricks Runtime 18 LTS release notes
• Databricks Runtime maintenance updates