Home / Databricks on Azure

Databricks on AzureTroubleshootingIntermediate

Long-running Databricks jobs reading Avro hit \"GC overhead limit exceeded\" - the DBR 18 Avro fast reader known issue and workaround

Databricks Runtime 18 LTS lists a known issue where the Avro fast reader slowly exhausts executor heap in long-running jobs, ending in OutOfMemoryError GC overhead limit exceeded. Confirm the pattern, then disable the fast reader on driver and executors and restart the cluster.

Deepak Bhardwaj3 min read

Applies toAzure Databricks Runtime 18 LTS; jobs and streams that read Avro files or Avro-encoded data over long periods

The known issue

From the Databricks Runtime 18 LTS release notes, Known issues:

  • In long-running jobs, the Avro fast reader (org.apache.avro.fastread) can exhaust executor heap.
  • The failure is java.lang.OutOfMemoryError: GC overhead limit exceeded.
  • Workaround: add -Dorg.apache.avro.fastread=false to both spark.driver.extraJavaOptions and spark.executor.extraJavaOptions (appending to any values already there), then restart the cluster.

Does it match your problem?

The pattern to look for:

Signal This issue Ordinary OOM
When it fails After hours or days of running fine Early, on a big partition or skewed key
Executor memory over time Climbs steadily across micro-batches or tasks Spikes on specific stages
Data format Avro (files, or Avro payloads decoded with from_avro) Any
Restart Fixes it for a while Doesn't help

Check executor memory in the Spark UI (Executors tab) or cluster metrics across the job's lifetime. A steady climb with frequent full GCs before the failure points here.

java.lang.OutOfMemoryError: GC overhead limit exceeded
  at org.apache.avro....

If the stack trace shows Avro decoding classes, that's a strong hint.

Apply the workaround

In the cluster's or job compute's Spark config:

spark.driver.extraJavaOptions -Dorg.apache.avro.fastread=false
spark.executor.extraJavaOptions -Dorg.apache.avro.fastread=false

If these options already have values (for example GC settings or a Java agent), append with a space; don't replace:

spark.executor.extraJavaOptions -XX:+UseG1GC -javaagent:/databricks/jars/agent.jar -Dorg.apache.avro.fastread=false

Then restart the cluster. JVM options only apply at startup.

For jobs, put this in the job cluster definition (Databricks Asset Bundles or Terraform), so new job clusters get it:

new_cluster:
  spark_version: 18.0.x-scala2.13   # use your exact LTS version string
  spark_conf:
    spark.driver.extraJavaOptions: "-Dorg.apache.avro.fastread=false"
    spark.executor.extraJavaOptions: "-Dorg.apache.avro.fastread=false"

Verify

  • In the Spark UI, Environment tab, check that both options contain the flag.
  • Watch executor memory over the same period that used to fail. It should level off.
  • Expect some loss of Avro decoding speed. Compare batch durations before and after.

When to remove it

Check the known issues section of the release notes after each maintenance update. Remove the flag once the issue is listed as fixed, and confirm the memory pattern stays flat.

If it doesn't help

Then it's likely a different memory problem:

  • Streaming state growing without a watermark
  • Caching DataFrames inside a loop without unpersisting
  • Collecting large results to the driver
  • Broadcasts that are too big

Checklist

  • Memory climbs steadily over time on an Avro workload
  • Flag added to driver and executor Java options, appended not replaced
  • Cluster restarted; flag visible in the Environment tab
  • Job definitions updated in IaC
  • Release notes checked after updates to remove the workaround

Sources

Related fixes