Kafka Schema Registry cheat sheet: subjects, compatibility checks and fixing rejected schemas

#When you need this

A producer deploy fails with Schema being registered is incompatible with an earlier schema (HTTP 409), a consumer throws Unknown magic byte!, or someone asks "which schema is this topic actually using?". Everything below uses the REST API with curl and jq, so it works the same on any host that can reach the registry.

Set these once:

export SR=https://schema-registry.example.internal:8081
export CT='Content-Type: application/vnd.schemaregistry.v1+json'
# Confluent Cloud or secured registries: add basic auth
export AUTH="-u $SR_API_KEY:$SR_API_SECRET"

Drop $AUTH if your registry has no authentication.

#Concepts in one table

Other subject strategies are RecordNameStrategy and TopicRecordNameStrategy, set on the client with value.subject.name.strategy. If the producer and the consumer use different strategies, they look in different subjects.

#Read: subjects, versions and IDs

# All subjects
curl -s $AUTH $SR/subjects | jq .

# Versions of one subject, and the latest schema
curl -s $AUTH $SR/subjects/orders-value/versions
curl -s $AUTH $SR/subjects/orders-value/versions/latest | jq .

# Just the schema text, pretty-printed
curl -s $AUTH $SR/subjects/orders-value/versions/latest/schema | jq .

# A schema by global ID (the ID from a message or an error)
curl -s $AUTH $SR/schemas/ids/1042 | jq -r .schema | jq .

# Which subjects/versions use this ID
curl -s $AUTH $SR/schemas/ids/1042/versions | jq .

# Include soft-deleted subjects and versions
curl -s $AUTH "$SR/subjects?deleted=true"
curl -s $AUTH "$SR/subjects/orders-value/versions?deleted=true"

#Find the schema ID inside a message

kcat -C -b broker1:9092 -t orders -o -1 -c 1 -e -f '%s' | head -c 5 | xxd
# 00000000: 00 00 00 04 12   -> magic byte 0, schema ID 0x00000412 = 1042

If the first byte is not 00, the message was not written with a Schema Registry serializer. That is the cause of Unknown magic byte! on the consumer: usually a producer writing plain JSON or bytes to a topic consumers expect to be Avro.

#Is this exact schema already registered?

jq -n --rawfile s order-v5.avsc '{schemaType:"AVRO", schema:$s}' > req.json
curl -s $AUTH -X POST -H "$CT" --data @req.json $SR/subjects/orders-value | jq .

Returns the subject, version and ID if it exists, or 40403 Schema not found. Omit schemaType for Avro, or use PROTOBUF / JSON.

#Test compatibility before you deploy

Run this in CI against the production registry (read-only call) before merging a schema change:

curl -s $AUTH -X POST -H "$CT" --data @req.json \
  "$SR/compatibility/subjects/orders-value/versions/latest?verbose=true" | jq .
# {"is_compatible": false, "messages": ["...READER_FIELD_MISSING_DEFAULT_VALUE..."]}

verbose=true adds the reason on current versions; older registries ignore it and return only is_compatible. With a _TRANSITIVE setting, test against all versions, not just latest; the registry does that when the subject is transitive.

#Compatibility settings

# Global default
curl -s $AUTH $SR/config

# Subject override (40401/40408 means none is set - the global applies)
curl -s $AUTH $SR/config/orders-value
curl -s $AUTH "$SR/config/orders-value?defaultToGlobal=true"

# Set a subject's compatibility
curl -s $AUTH -X PUT -H "$CT" --data '{"compatibility":"BACKWARD_TRANSITIVE"}' \
  $SR/config/orders-value

# Remove the override and fall back to global
curl -s $AUTH -X DELETE $SR/config/orders-value

What each rule lets you change (Avro):

Renaming a field or changing its type is breaking under every rule except NONE. Avro aliases and type promotions (for example int to long) are allowed in some cases; test them with the compatibility call rather than assuming.

#Fixing a 409 rejection

#Deliberate breaking change on one subject

Warning

Every consumer still on the old schema may fail to deserialize, or silently read wrong values, once producers switch. Only do this with all consumer owners on the change, and prefer a new topic for anything customer-facing.

curl -s $AUTH -X PUT -H "$CT" --data '{"compatibility":"NONE"}' $SR/config/orders-value
curl -s $AUTH -X POST -H "$CT" --data @req.json $SR/subjects/orders-value/versions
curl -s $AUTH -X PUT -H "$CT" --data '{"compatibility":"BACKWARD"}' $SR/config/orders-value

Never change the global setting to get one subject through.

#Freeze the registry during an incident or migration

curl -s $AUTH $SR/mode
curl -s $AUTH -X PUT -H "$CT" --data '{"mode":"READONLY"}' $SR/mode/orders-value
curl -s $AUTH -X PUT -H "$CT" --data '{"mode":"READWRITE"}' $SR/mode/orders-value

READONLY stops a misbehaving producer from registering more bad versions while you investigate. Producers that need to register will fail, which is usually what you want.

#Deleting versions and subjects

Caution

A hard delete removes the schema ID mapping. Any message still in Kafka that carries that ID can no longer be deserialized. Check topic retention before hard-deleting anything referenced by data you still keep.

# Soft delete one version (recoverable, ID still resolves)
curl -s $AUTH -X DELETE $SR/subjects/orders-value/versions/6

# Soft delete the whole subject
curl -s $AUTH -X DELETE $SR/subjects/orders-value

# Hard delete - only after a soft delete
curl -s $AUTH -X DELETE "$SR/subjects/orders-value/versions/6?permanent=true"
curl -s $AUTH -X DELETE "$SR/subjects/orders-value?permanent=true"

To undo a bad registration, soft-delete that version: the next compatibility check compares against the previous one. Don't hard-delete to "fix" a 409.

#Error codes you will see

#Prevent it next time

• Register schemas from CI, not from producers at runtime, and run the compatibility call as a pipeline gate.
• Prefer BACKWARD_TRANSITIVE or FULL_TRANSITIVE for long-retention topics, so a new version is compatible with every version still on disk.
• Give every new optional field a default from day one.
• On self-managed registries, the schemas live in the compacted _schemas topic (name configurable with kafkastore.topic). Replicate or back it up alongside the cluster; without it, no Avro message can be read.
• Alert on 409 and 40401 rates in producer logs; a spike usually means an unreviewed deploy.