DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

OrderHub Day 33: a schema registry moves contract failures left — from a 500 in a consumer to a red build in CI

#kafka #backend #java #springboot

The events flowing between services are an API just as real as the REST endpoints — but on OrderHub so far they've been plain JSON blobs on a Kafka topic, and a JSON blob is a weak contract. Nothing stops a producer renaming item to sku, flipping quantity from int to string, or dropping a field. The message serializes fine, lands on the topic, and the break only surfaces later — as a JsonMappingException in a consumer, at runtime, in production, in a different service from the one that made the change. Day 33 fixes that with Avro and a schema registry. I built an interactive page that publishes one OrderPlaced event through the whole pipeline so you can see exactly who catches a breaking change, and when.

Avro: schema-first, compiled to a typed record

Avro flips the model. You declare the event's shape once, in a .avsc file, and that schema is the single source of truth. The avro-maven-plugin compiles it into a generated SpecificRecord at build time, so there's no hand-written, drift-prone POJO — the Java type and the wire schema are generated from the same declaration and can't disagree:

{
  "namespace": "dev.dev48v.orderhub.avro",
  "type": "record", "name": "OrderPlaced",
  "fields": [
    { "name": "eventId",  "type": "string" },
    { "name": "orderId",  "type": "string" },
    { "name": "item",     "type": "string" },
    { "name": "quantity", "type": "int"    },
    { "name": "status",   "type": "string" },
    { "name": "placedAt", "type": "long" }, { "name": "occurredAt", "type": "long" }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Avro's encoding is compact binary driven by that schema, so the bytes carry only the values — much smaller than JSON, which re-sends every field name on every message.

The registry: subjects, versions, and an id on the wire

A schema registry is a small service that stores schemas and hands each one a numeric id. Schemas are grouped under a subject (by default <topic>-value, e.g. order-placed-avro-value), and each subject accumulates versions as the schema evolves. On publish, Confluent's KafkaAvroSerializer registers the writer schema, gets back an id, and puts a one-byte magic + 4-byte id + the compact binary on the topic — not the schema, and not the field names, on every message:

// on the wire (Confluent wire format):
//   [ 0x00 ][ 4-byte schema id ][ Avro binary payload ]
//   e.g.  0x00  00 00 00 2A  <binary>     (id = 42)
Enter fullscreen mode Exit fullscreen mode

The consumer reads the id, fetches the matching schema from the registry (cached after the first lookup), and — with specific.avro.reader=true — decodes straight into the generated OrderPlaced, so a @KafkaListener takes a typed event with real getters, exactly like the JSON path did.

The whole point: compatibility gates evolution

Here's where the registry earns its keep. Before it stores a new version of a subject's schema, it runs a compatibility check against the existing one. The default rule is BACKWARD: a consumer on the new schema must be able to read data written with the old one. Under that rule:

  • Add an optional field with a default → compatible. Old records read back with the default filled in, so it registers as v2 and both old and new consumers keep working.
  • Add a required field with no default → incompatible. A new reader would demand a field old records don't have and can't default, so the registry rejects the registration with a 409 and the serializer fails the publish. 
// v1 -> add OPTIONAL field WITH default  => BACKWARD-compatible
{ "name":"channel", "type":"string", "default":"WEB" }   // old data reads as "WEB"

// v1 -> add REQUIRED field, NO default   => INCOMPATIBLE -> publish REJECTED
{ "name":"channel", "type":"string" }
Enter fullscreen mode Exit fullscreen mode

The breaking change is caught at publish time, in the producer's pipeline — never as a 3 a.m. page from a consumer.

What the demo actually shows

Pick an encoding (Avro + registry, or ad-hoc JSON) and a schema change (v1, +optional, +required), then hit Publish. The event walks producer → registry → topic → consumer, and you watch where it stops. Under Avro, the +required change is rejected at the registry — it never reaches Kafka, and every consumer is protected. Under JSON there's no gate, so the same change sails onto the topic and detonates in the consumer at runtime. The +optional change, by contrast, evolves cleanly under Avro: only an id + binary travels, and older records get channel="WEB" from the default.

The page mirrors the real code, which is gated behind orderhub.avro.enabled (default false) so the Day-25 JSON path is untouched when off. One @EmbeddedKafka test round-trips an event through a mock:// registry — no external registry, no Docker — and an evolution test proves the optional-with-default field is BACKWARD compatible while the required-no-default field is rejected. Same events, but now a breaking contract change is a build error, not a production outage.

Publish the event and watch who catches the break:
https://dev48v.infy.uk/orderhub.php

Repo: https://github.com/dev48v/order-hub-from-zero

Top comments (0)