Sev1: a producer registered an incompatible schema and Kafka consumers crash on deserialization

#Symptoms

• Consumers log org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id 1234, Schema not found, or field-type errors (ClassCastException, missing required field).
• Consumer groups stop at one offset per partition and lag climbs.
• A producer service was deployed shortly before.

#Step 1: confirm a schema change (5 minutes)

SR=https://schema-registry.internal:8081
SUBJECT=orders-value

curl -s $SR/subjects/$SUBJECT/versions
curl -s $SR/subjects/$SUBJECT/versions/latest | jq '{version, id, schema: (.schema|fromjson)}'
curl -s $SR/config/$SUBJECT; echo; curl -s $SR/config; echo

If the latest version appeared at the time of the deploy, and the subject or global compatibility is NONE (or someone changed it), you have found the cause.

Check what changed:

curl -s $SR/subjects/$SUBJECT/versions/12 | jq -r .schema | jq . > v12.json
curl -s $SR/subjects/$SUBJECT/versions/13 | jq -r .schema | jq . > v13.json
diff v12.json v13.json

Typical breaking changes: a field removed without a default, a type changed (int to string), a field renamed, a required field added without default.

#Step 2: stop the bad producer

Roll back the producer release, or scale it to zero. Every minute it runs writes more records the consumers cannot read.

kubectl -n orders rollout undo deploy/order-service

#Step 3: get consumers moving

The bad records are already in the topic. Choose per consumer group:

Spring Kafka example: ErrorHandlingDeserializer wrapping the Avro deserializer, with a DeadLetterPublishingRecoverer. Kafka Connect: errors.tolerance=all with errors.deadletterqueue.topic.name.

Skipping offsets (only when you have agreed the records can be lost or replayed):

kafka-consumer-groups.sh --bootstrap-server $BS --group billing --describe
kafka-consumer-groups.sh --bootstrap-server $BS --group billing \
  --topic orders:3 --reset-offsets --to-offset 884213 --execute

Stop the group's consumers before resetting; offsets can only be reset for an inactive group.

Caution

Skipped records are lost to that consumer. Record exactly which partitions and offset ranges you skipped so they can be replayed after a fix.

#Step 4: fix the registry

Delete the bad version (soft delete) so producers with auto-registration do not keep using it, and restore compatibility:

curl -s -X DELETE $SR/subjects/$SUBJECT/versions/13
curl -s -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "BACKWARD"}' $SR/config/$SUBJECT

Soft-deleting a version does not delete its schema ID; records already written with ID 1234 can still be read by consumers that know how. Do not hard-delete (?permanent=true) while records with that ID are in the topic.

#Step 5: replay the records you skipped (if any)

Once consumers can handle the data (new code, or a corrected producer re-publishing), replay the recorded offset ranges, or ask the producer team to re-publish the affected business events with the corrected schema.

#Step 6: confirm

• Consumer lag back to normal; no deserialization errors.
• Latest subject version is the intended one; compatibility is BACKWARD (or your standard).
• DLQ count known and owned by a team.

#Compatibility levels in one table

#Prevent it next time

• Producers set auto.register.schemas=false in production; schemas are registered by CI after a compatibility check:

curl -s -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data @new-schema-request.json $SR/compatibility/subjects/$SUBJECT/versions/latest
# {"is_compatible": true}

• Lock compatibility at the global level (BACKWARD or FULL_TRANSITIVE) and restrict who can change config via registry RBAC.
• Consumers use an error-handling deserializer with a DLQ, so one bad record never stops a group.
• Alert on consumer lag growth and on new subject versions in production.