Why now
LTS runtimes have an end-of-support date. After that, no fixes — and jobs on unsupported runtimes may be blocked from starting in future. Check the support lifecycle page for your versions.
1. Inventory
SELECT dbr_version, count(DISTINCT cluster_id) AS clusters
FROM system.compute.clusters
WHERE delete_time IS NULL
GROUP BY dbr_version ORDER BY clusters DESC;
Add job owners and the jobs using each cluster definition. System table columns vary by release — check DESCRIBE system.compute.clusters.
2. Read the release notes between versions
Common breaking areas when jumping several LTS releases:
| Area | What changes |
|---|---|
| Spark major/minor | Behaviour changes (ANSI mode, datetime parsing, CAST errors), removed APIs |
| Python | Python minor version → library compatibility, pinned wheels |
| Pre-installed libraries | Versions of pandas, numpy, scikit-learn change |
| Scala | 2.12 vs 2.13 for JAR jobs |
| Access mode | Some features only on UC access modes in newer runtimes |
| Defaults | Delta table features enabled by default (deletion vectors, etc.) — older readers may not read them |
Warning
New Delta table features (e.g. deletion vectors) can make tables unreadable by older clients (old runtimes, external readers). Upgrade readers first or keep features off on shared tables.
3. Test with shadow runs
For each job: clone it, set the new runtime, point outputs at a test schema, run on recent input, and compare row counts, checksums and runtime with the production run:
SELECT 'prod' src, count(*), sum(hash(*)) FROM prod.sales.daily WHERE run_date = '2026-10-24'
UNION ALL
SELECT 'test', count(*), sum(hash(*)) FROM test_upgrade.sales.daily WHERE run_date = '2026-10-24';
4. Fix libraries
Pin library versions in job specs; rebuild wheels for the new Python version; replace deprecated libraries. Use %pip freeze on old and new clusters to see differences.
5. Roll out by wave
- Low-risk jobs; 2. business-critical jobs with owners on standby; 3. all-purpose clusters (announce to users).
Update cluster policies to allow only the new runtime (or a short list) after each wave, so new jobs don't start on the old one.
6. Rollback
Keep the previous runtime allowed in the policy until each wave is stable; rollback is just changing spark_version back — as long as no table features were enabled that the old runtime can't read.
Checklist
- Runtime inventory with owners
- Breaking changes reviewed per jump
- Shadow runs compared
- Libraries pinned and rebuilt
- Waves done; policies updated