What changed
The Databricks Runtime maintenance update dated 24 July 2026 (18 LTS and 18.2) says that custom or third-party types used in DStream closures must now be listed before checkpoint recovery:
Users must now set
spark.streaming.checkpoint.allowedClassesto those types before checkpoint recovery.
DStream checkpoints contain serialised Java objects (the DStream graph and the closures in it). Restricting which classes can be deserialised from a checkpoint closes a class of deserialisation attacks, where a tampered checkpoint file could run arbitrary code. The cost is that jobs relying on custom types need an explicit allow-list.
Symptoms
- A DStream job that has run for months fails on restart, typically right after a cluster restart picked up the maintenance update.
- The failure happens during
StreamingContext.getOrCreate(checkpointDir, ...), with an error about a class not being allowed or failing to deserialise. - Starting with a new checkpoint directory works, which tells you the problem is recovery, not the job logic.
Don't "fix" it by deleting the checkpoint. You'd lose the job's progress and may reprocess or skip data.
1. Find the classes
The error message usually names the class it refused. Collect all of them:
- Case classes, POJOs and enums used inside
map,updateStateByKey,mapWithState,foreachRDDand similar closures. - Third-party types captured in closures (for example a JSON parser, a Kafka record wrapper, a protobuf message class).
- Scala/Java collection types from libraries other than the standard library.
Running the recovery in a test cluster against a copy of the checkpoint directory lets you iterate without touching production.
2. Set the allow-list
In the cluster or job's Spark config:
spark.streaming.checkpoint.allowedClasses com.example.events.ClickEvent,com.example.events.SessionState,org.thirdparty.Codec
Check the release notes (and test) for the exact format the setting accepts on your runtime, for example whether it takes a comma-separated list of fully qualified class names and whether packages or wildcards are supported. Keep the list as narrow as possible; allowing broad packages weakens the protection.
Set it at cluster/job level, not in notebook code after the context starts, because it must be in place before recovery.
3. Lock it down
- Store the setting in the job definition (Databricks Asset Bundles or Terraform) so it survives cluster recreation.
- Protect the checkpoint location: only the job's identity should be able to write to it. The allow-list is a second line of defence, not a replacement for access control.
4. Plan the move off DStreams
DStreams (the StreamingContext API) are legacy. Structured Streaming checkpoints store offsets and state in a documented format rather than serialised closures, and they're what Databricks develops further. For Kafka or Event Hubs sources, a typical migration:
| DStream | Structured Streaming |
|---|---|
KafkaUtils.createDirectStream | spark.readStream.format("kafka") |
updateStateByKey / mapWithState | transformWithState or applyInPandasWithState |
foreachRDD writes | foreachBatch or a native sink (Delta) |
| Checkpoint directory | checkpointLocation option per query |
See also Structured Streaming checkpoint recovery.
Checklist
- Failure confirmed as checkpoint recovery (new checkpoint works)
- Classes identified from errors and closure code
-
spark.streaming.checkpoint.allowedClassesset narrowly at job level - Setting stored in IaC / bundle definitions
- Checkpoint location write access restricted
- Migration to Structured Streaming scheduled