Gowtham Potureddi

Posted on Aug 21

Kafka Schema Registry Deep Dive: Confluent vs Apicurio vs Glue — Compatibility & Governance

#python #sql #interview #dataengineering

A Kafka Schema Registry is the contract layer that decides whether a producer team can rename a field on Tuesday without paging every consumer team on Wednesday — and it is the component that separates a Kafka platform which scales to hundreds of topics and dozens of teams from one that collapses into a graveyard of undocumented byte blobs. Every record a producer serialises to a topic carries a decade of downstream assumptions: a fraud model reads amount_cents, a warehouse sink expects event_time as a logical timestamp, a search indexer tolerates a missing nickname only because it has a default. Ship a producer that drops a required field or changes an int to a string, and those consumers deserialize garbage or crash — unless a registry sat between the two teams and refused to register the incompatible schema in the first place. The engineering decision is not "should we use schemas" — any stream with more than one consumer needs them — but which registry you run, what compatibility mode you enforce, and who owns each subject.

This guide is the senior-data-engineering walkthrough for the schema layer, framed the way interviewers probe it: the compatibility modes (BACKWARD, FORWARD, FULL, and their TRANSITIVE variants) and the Avro schema evolution rules that make an add-a-field change safe and a remove-a-required-field change breaking; the Confluent Schema Registry wire format and subject naming strategies; the two main alternatives — Apicurio (open-source, Confluent-compatible) and AWS Glue Schema Registry (AWS-native, IAM-governed); and the governance workflow — CI compatibility gates, ownership, and access control — that stops a breaking change ever reaching production. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse on the event-processing practice library →, and sharpen the schema axis with the JSON practice library →.

On this page

Why the schema registry is the contract layer of a Kafka platform
Compatibility modes and schema evolution
Confluent Schema Registry deep dive
Apicurio and AWS Glue Schema Registry
Governance, ownership, and CI enforcement
Cheat sheet — Kafka Schema Registry recipes
Frequently asked questions
Practice on PipeCode

1. Why the schema registry is the contract layer of a Kafka platform

The registry stores writer schemas by subject and puts an ID on the wire — that indirection is what lets producers and consumers evolve independently

The one-sentence invariant: a Kafka Schema Registry is a versioned, compatibility-checked store of writer schemas keyed by subject, and because the producer writes only a small schema ID onto the wire (not the schema itself), the consumer can fetch the exact writer schema by ID and resolve it against its own reader schema — which is precisely the indirection that lets two teams deploy on different days without breaking each other, provided a compatibility mode gates every new schema version. Remove the registry and you are back to either shipping the whole schema with every message (enormous overhead) or, worse, an implicit contract living only in tribal knowledge that breaks silently at 3 AM.

The four axes interviewers actually probe.

Serialization format. Avro, Protobuf, or JSON Schema. Avro is the historical default because its resolution rules (writer schema plus reader schema) map cleanly onto the registry model; Protobuf and JSON Schema are first-class in modern Confluent and Apicurio. The format decides how field defaults, enums, and type changes behave under evolution. Interviewers open here because the compatibility rules differ subtly per format.
Compatibility mode. BACKWARD (default in Confluent), FORWARD, FULL, NONE, and the TRANSITIVE variants. The mode decides which changes are legal and, critically, which side upgrades first. Getting this wrong ships a rollout order that deadlocks — the classic "we deployed consumers first but the mode only protects producer-first" trap.
Subject-naming strategy. TopicNameStrategy (one schema per topic), RecordNameStrategy (one subject per record type, many types per topic), or TopicRecordNameStrategy. The strategy decides whether a topic can carry multiple event types and how compatibility is scoped. Senior signal is knowing this is configurable, not fixed.
Governance and ownership. Who is allowed to register a new version? How is a breaking change stopped before it hits prod? Who owns the subject when producer and consumer are different teams? The registry is a shared contract, so this is an organisational question as much as a technical one.

The 2026 reality — three registries, one wire contract.

Confluent Schema Registry is the reference implementation and market default. It backs schemas in a compacted _schemas Kafka topic, exposes a REST API, and its client serializers define the de-facto wire format (magic byte + 4-byte schema ID). If you can pick and you are not AWS-locked, you pick Confluent.
Apicurio Registry is the open-source, Apache-2.0 alternative from Red Hat. It stores artifacts in groups, supports Avro/Protobuf/JSON Schema/OpenAPI/AsyncAPI, and — crucially — ships a Confluent-compatible REST API (ccompat) so existing Confluent serializers work against it with only a URL change.
AWS Glue Schema Registry is the AWS-native option. Schemas live under registry → schema → version, compatibility is set on the registry, and access is governed by IAM rather than a bespoke ACL system. It integrates natively with MSK, Kinesis, and Lambda; its serializers use a different wire header (a UUID schema-version-id, not a 4-byte int).

What interviewers listen for.

Do you say "the wire carries a schema ID, not the schema" in the first minute? — required answer.
Do you distinguish writer schema from reader schema and name Avro's resolution? — senior signal.
Do you connect the compatibility mode to the deployment order (consumers-first vs producers-first)? — senior signal.
Do you name the subject-naming strategy as the lever for multi-event topics? — senior signal.
Do you frame the registry as a shared contract with an owner, not just a storage service? — required answer.

Worked example — the four-axis comparison table

Detailed explanation. The single most useful artifact for a schema-registry interview is a memorised comparison across the three registries and the four axes. Every senior discussion converges on it within ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical payments.transactions topic feeding a fraud model, a warehouse sink, and a ledger service.

Topic. payments.transactions, one Avro record type, ~50k msg/s.
Consumers. Fraud model (needs every field, deploys weekly), Snowflake sink (tolerant, deploys monthly), ledger service (strict, deploys rarely).
Constraint. No consumer can be forced to redeploy in lockstep with the producer.
Cloud. Runs on self-managed Kafka today; an AWS MSK migration is on the roadmap.

Question. Build the registry-and-axis comparison and pick the registry, format, compatibility mode, and subject strategy for payments.transactions.

Input.

Axis	Confluent	Apicurio	AWS Glue
Storage	`_schemas` Kafka topic	groups + artifacts (SQL/KV)	registry → schema → version
Wire header	magic byte + 4-byte int ID	same (via ccompat)	UUID schema-version-id
Access control	RBAC / ACLs	roles + content rules	IAM
Default compat	BACKWARD	BACKWARD (configurable)	BACKWARD (configurable)

Code.

Decision for payments.transactions
===================================
Format:            Avro   (mature resolution rules; fraud + ledger both Avro-native)
Registry:          Confluent now; Apicurio ccompat keeps the door open, Glue after MSK
Compatibility:     BACKWARD  (new schema readable against old data -> consumers upgrade first-safe)
Subject strategy:  TopicNameStrategy  (single record type on the topic)
Subject name:      payments.transactions-value
Owner:             payments-platform team (producer) owns the subject
Consumers:         subscribe to the contract; no lockstep redeploys

Step-by-step explanation.

Avro wins the format axis because both the fraud model and the ledger service already speak Avro and its writer/reader resolution is the best-documented; Protobuf would work but adds a second toolchain.
Confluent is the primary registry because the platform is self-managed today, but choosing Avro plus the Confluent serializer keeps the wire contract portable — an Apicurio ccompat endpoint or a later Glue move is a client-config change, not a re-serialisation.
BACKWARD compatibility is chosen because the dominant risk is a producer racing ahead of slow consumers; BACKWARD guarantees a new schema can still read data written under the old schema, so consumers can upgrade on their own schedule after the producer.
TopicNameStrategy fits because the topic carries exactly one record type — the subject is simply payments.transactions-value, and compatibility is scoped to that single evolving schema.
Ownership is assigned to the producing team; consumers subscribe to the published contract. This is the organisational half of the answer and the one weak candidates omit.

Output.

Decision	Choice	Why
Format	Avro	Best-documented resolution; both strict consumers are Avro-native
Registry	Confluent (Apicurio/Glue portable)	Self-managed today; keep interop open for MSK
Compatibility	BACKWARD	Producer-can-lead; consumers upgrade on their own schedule
Subject strategy	TopicNameStrategy	One record type per topic
Owner	Producing team	Contract has a single accountable owner

Rule of thumb. Never pick a registry on brand familiarity. Pick it on (format × compatibility × subject-strategy × governance) and on cloud lock-in. Write the four-axis table first; the registry falls out of the constraints.

Worked example — what interviewers actually probe

Detailed explanation. The senior schema-registry interview has a predictable shape: an ambiguous opener ("how do you stop producers and consumers breaking each other on Kafka?"), then progressive narrowing to test whether you know the axes. Candidates who say "schema registry with a BACKWARD compatibility gate" in sentence one score highest; candidates who say "we document the schema in a wiki" score lowest. Walk through the grading rubric.

Ambiguous opener. "How do teams share a Kafka topic safely?" — invites you to name the registry and a compatibility mode.
Follow-up 1. "A producer adds a field — does it break consumers?" — probes evolution rules.
Follow-up 2. "A producer removes a required field — what happens?" — probes the illegal-change case.
Follow-up 3. "Who deploys first, producer or consumer?" — probes the compatibility-direction link.
Follow-up 4. "How do you stop the breaking change reaching prod?" — probes governance.

Question. Draft a 5-minute senior answer that covers all four axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Mechanism named	"we agree on a JSON shape"	"a schema registry stores writer schemas; the wire carries a schema ID"
Add a field	"it might break"	"safe under BACKWARD if the new field has a default"
Remove a field	"should be fine"	"removing a required field breaks BACKWARD; needs a default first"
Deploy order	"doesn't matter"	"BACKWARD = consumers can upgrade after producers; FORWARD = the reverse"
Enforcement	"code review"	"CI compatibility check before register; registry rejects incompatible versions"

Code.

Senior schema-registry answer template (5 minutes)
==================================================

Minute 1 — name the mechanism
  "Put a schema registry between producers and consumers. The producer
   registers a writer schema, gets an ID, and writes only the ID on the
   wire. The consumer fetches the writer schema by ID and resolves it
   against its own reader schema."

Minute 2 — evolution rules
  "Adding a field with a default is a backward-compatible change: a new
   consumer reading old data fills the default. Removing a REQUIRED
   field or changing a type is breaking. The registry enforces this per
   compatibility mode."

Minute 3 — compatibility direction
  "BACKWARD (the default) means a new schema can read data written by the
   previous schema, so consumers can be upgraded AFTER producers.
   FORWARD is the mirror — old consumers read new data — so you upgrade
   consumers FIRST. FULL is both. TRANSITIVE checks against ALL prior
   versions, not just the last one."

Minute 4 — subject naming + format
  "Subject naming decides scope: TopicNameStrategy = one schema per
   topic; RecordNameStrategy = many event types per topic keyed by record
   name. Format is Avro/Protobuf/JSON Schema; Avro's resolution rules are
   the reference."

Minute 5 — governance
  "Compatibility is checked in CI before the schema is registered, so a
   breaking PR fails at review time, not in production. The producing
   team owns the subject; access is controlled by RBAC/IAM."

Step-by-step explanation.

Minute 1 frames the whole answer: naming the ID-on-the-wire indirection immediately signals you understand why the registry exists, not just that it exists.
Minute 2 grounds the abstraction in the two canonical changes every interviewer asks about — add a field (safe with a default) and remove a required field (breaking). Naming the default requirement is the senior tell.
Minute 3 is the axis most candidates miss: compatibility mode dictates deployment order. Saying "BACKWARD → producers first, consumers after" out loud preempts the follow-up.
Minute 4 shows breadth — subject naming and format are levers, not fixed. Mentioning RecordNameStrategy signals you have run multi-event topics.
Minute 5 closes the loop with governance: the registry rejects bad schemas, but the cheap place to catch them is CI, before the producer ever calls register.

Output.

Grading criterion	Weak score	Senior score
Names ID-on-the-wire indirection	rare	mandatory
Names default-value rule for add-field	occasional	mandatory
Links compatibility to deploy order	rare	senior signal
Names a subject-naming strategy	rare	senior signal
Names CI compatibility gate	rare	senior signal

Rule of thumb. The senior schema answer is a 5-minute monologue that covers format, compatibility direction, subject naming, and the CI gate without waiting for the follow-ups. Rehearse it once; deploy it every time.

Worked example — the "pick the registry" decision tree

Detailed explanation. Given a new Kafka platform, the senior architect runs a short decision tree in their head. Codifying it makes the interview answer reproducible: any interviewer can hand you a scenario and you can walk the tree out loud. Walk the tree with three canonical scenarios — a self-managed on-prem cluster, an all-in AWS MSK shop, and a multi-cloud team that wants no vendor lock.

Q1. Are you fully on AWS (MSK / Kinesis / Lambda) and happy with IAM governance? → yes = Glue; no = Q2.
Q2. Do you need open-source / self-host / no per-schema licensing, or multi-format (OpenAPI/AsyncAPI too)? → yes = Apicurio; no = Q3.
Q3. Default → Confluent Schema Registry (reference implementation, richest ecosystem).
Q4 (parallel). Whatever you pick, keep the Confluent wire contract (Avro + magic-byte+ID) so an Apicurio ccompat swap stays a client-config change.

Question. Walk the decision tree for the three scenarios and record the registry each ends up with.

Input.

Scenario	Q1 (AWS-native?)	Q2 (open/multi-format?)	Q4 (keep interop?)
On-prem self-managed	no	maybe	yes
All-in AWS MSK	yes	—	yes
Multi-cloud, no lock-in	no	yes	yes

Code.

# Decision-tree helper (illustrative)
def pick_registry(aws_native: bool,
                  needs_open_source: bool,
                  keep_wire_interop: bool = True) -> dict:
    """Return the primary registry choice and interop note."""
    if aws_native:
        registry = "AWS Glue Schema Registry"
    elif needs_open_source:
        registry = "Apicurio Registry"
    else:
        registry = "Confluent Schema Registry"

    return {
        "registry": registry,
        "wire": "Avro + magic-byte + schema-id (Confluent-style)"
                if keep_wire_interop else "registry-native",
    }


print(pick_registry(False, False))
# -> {'registry': 'Confluent Schema Registry', 'wire': 'Avro + magic-byte + schema-id (Confluent-style)'}

print(pick_registry(True, False))
# -> {'registry': 'AWS Glue Schema Registry', 'wire': 'Avro + magic-byte + schema-id (Confluent-style)'}

print(pick_registry(False, True))
# -> {'registry': 'Apicurio Registry', 'wire': 'Avro + magic-byte + schema-id (Confluent-style)'}

Step-by-step explanation.

Scenario 1 — on-prem, self-managed Kafka, no AWS pull and no hard open-source mandate. Q1 = no, Q2 = maybe (not required) → Confluent, the reference implementation with the richest client and connector ecosystem.
Scenario 2 — all-in AWS MSK, comfortable governing schemas through IAM. Q1 = yes → Glue, because the serializers, MSK integration, and IAM policy model are AWS-native and remove a separate service to operate.
Scenario 3 — multi-cloud with an explicit no-lock-in rule and a wish for OpenAPI/AsyncAPI artifacts alongside Avro. Q2 = yes → Apicurio, Apache-2.0, self-hostable anywhere, multi-format.
The parallel Q4 branch keeps every choice portable: standardising on the Avro-plus-Confluent-wire contract means the client only changes a URL to move between Confluent and Apicurio ccompat. Glue uses a different header, so a Glue move is a serializer swap, not just a URL — plan for that.
If none of Q1/Q2 dominate, Confluent is the safe default; the ecosystem breadth outweighs the license consideration for most teams.

Output.

Scenario	Registry	Interop note
On-prem self-managed	Confluent	Portable wire; Apicurio ccompat later is a URL change
All-in AWS MSK	Glue	IAM-governed; serializer swap, different wire header
Multi-cloud, no lock-in	Apicurio	Open-source, multi-format, Confluent-compatible API

Rule of thumb. The decision tree is AWS-native → open-source → default. Whatever you pick, standardise on the Avro + Confluent wire contract so the registry stays a config choice, not a re-serialisation project.

Senior interview question on schema-registry design

A senior interviewer often opens with: "You are standing up a shared Kafka platform for eight teams. Producers and consumers deploy independently and nobody trusts a wiki page to keep schemas in sync. Walk me through the registry you would introduce, the compatibility policy, the subject-naming choice, and how you stop one team's producer from breaking another team's consumer."

Solution Using a Confluent registry with BACKWARD compatibility and a CI gate

# 1. Producer serializer config (Kafka client)
# Avro record serialised with the schema ID on the wire
schema.registry.url: http://schema-registry:8081
key.serializer:   org.apache.kafka.common.serialization.StringSerializer
value.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
# auto-register in dev only; prod registers via CI (use.latest.version=true)
auto.register.schemas: false
use.latest.version: true

# 2. Set the platform-wide default compatibility to BACKWARD, then tighten
# per-subject where a stricter FULL is warranted.
curl -X PUT http://schema-registry:8081/config \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "BACKWARD"}'

# 3. Register the value subject for payments.transactions
curl -X POST http://schema-registry:8081/subjects/payments.transactions-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d @transactions-v1.avsc.json

# 4. CI compatibility gate — runs on every schema PR, BEFORE register
# Returns is_compatible:false and fails the build if the change breaks BACKWARD.
curl -s -X POST \
  http://schema-registry:8081/compatibility/subjects/payments.transactions-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d @transactions-v2.avsc.json | jq -e '.is_compatible == true'

Step-by-step trace.

Step	Before (wiki-documented schema)	After (registry + CI gate)
Contract location	wiki page, drifts silently	`_schemas` topic, versioned
Wire payload	JSON blob, full field names	magic byte + 4-byte schema ID + Avro
Add-field change	hope consumers cope	allowed only if it keeps BACKWARD
Remove-required change	ships, breaks consumers	rejected by registry / fails CI
Deploy order	ad-hoc	producers first, consumers after (BACKWARD)
Breaking change caught	in production, at 3 AM	in the PR, at review time

After the rollout, every producer registers a compatibility-checked writer schema and writes only a schema ID; consumers fetch the writer schema by ID and resolve it against their reader schema; a PR that would drop amount_cents fails the CI compatibility call before it can merge, so the break never reaches a running consumer.

Output:

Metric	Before	After
Broken deploys per quarter	3–4	~0
Wire payload size	full JSON	ID + binary Avro (smaller)
Schema source of truth	wiki	`_schemas` topic
Breaking-change detection	prod incident	CI, pre-merge
Cross-team coordination	lockstep releases	independent deploys

Why this works — concept by concept:

Schema ID on the wire — the producer registers the writer schema once and embeds a 4-byte ID per message; the consumer caches the schema by ID. This is the indirection that decouples the two teams' release schedules.
BACKWARD compatibility — the registry guarantees a new schema can still read data written under the previous schema, so consumers may lag the producer safely. It encodes the deployment order into the contract.
CI compatibility gate — the /compatibility endpoint answers "would this register?" without registering, so a breaking change fails the build instead of a consumer. Shifting the check left turns a prod incident into a red PR.
Producing-team ownership — one accountable owner per subject prevents the "everyone and no one owns it" drift that killed the wiki approach.
Cost — one registry service (or managed add-on), a _schemas topic, and a CI step per schema PR. The eliminated cost is the recurring cross-team broken-deploy incident and the lockstep-release coordination tax. O(1) per message on the wire; O(schema-versions) storage, which is tiny.

SQL
Topic — streaming
Streaming schema and contract problems

Practice →

SQL Topic — event-processing Event-processing pipeline problems

Practice →

2. Compatibility modes and schema evolution

`compatibility modes` are the rulebook the registry enforces on every new version — and they encode which side of the pipeline may upgrade first

The mental model in one line: a compatibility mode is a predicate the registry evaluates before accepting a new schema version — BACKWARD asks "can a consumer on the new schema read data written by the old schema?", FORWARD asks "can a consumer on the old schema read data written by the new schema?", FULL demands both, NONE disables the check, and the TRANSITIVE variants apply the predicate against every prior version rather than only the latest — and the mode you pick dictates whether producers or consumers must deploy first. Every senior engineer has shipped a "compatible" change that broke prod because they reasoned about the wrong direction.

The modes, precisely.

BACKWARD (Confluent default). New schema can read data written by the previous version. Legal changes: delete a field, add an optional field (one with a default). Deploy order: consumers first is safe — wait, invert it — because the new consumer must read old data, you upgrade consumers to the new schema and they can still read the backlog written by old producers. In practice teams register the new schema, upgrade consumers, then producers.
FORWARD. New schema data can be read by a consumer on the previous version. Legal changes: add a field, delete an optional field. Deploy order: producers first is safe — old consumers keep reading the new producer's data.
FULL. Both BACKWARD and FORWARD hold. Legal changes: only add/remove optional fields (fields with defaults). The safest and most restrictive; either side may deploy first.
NONE. No checking. Use only for a greenfield subject before the first consumer exists, or when you truly manage compatibility out of band. In prod this is a foot-gun.
*_TRANSITIVE. BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, FULL_TRANSITIVE check the new schema against all previous versions, not just the immediately preceding one. Non-transitive checks only the last version — which lets a chain of individually-compatible steps drift a v1 consumer into incompatibility.

Avro schema-evolution rules — the mechanics under the modes.

Add a field. Safe under BACKWARD only if the field has a default, because a new reader encountering old data (which lacks the field) fills the default. Without a default, a new reader cannot decode old records.
Remove a field. Safe under BACKWARD if the removed field had a default (the new reader ignores it); the mirror holds under FORWARD for the writer side.
Rename a field. Not directly compatible; use an Avro alias so the reader maps the old name to the new one.
Change a type. Only promotions are legal (e.g. int → long, float → double, int → float). Narrowing (long → int) is breaking. Changing string ↔ int is breaking.
Enum and union changes. Adding an enum symbol is backward-breaking unless the schema uses a default for unknown symbols (Avro 1.9+); widening a union is generally FORWARD-safe, narrowing is BACKWARD-safe.

The three failure modes senior engineers pre-empt.

Wrong direction. Reasoning about BACKWARD when the rollout is producer-first. Fix: write the direction on the whiteboard — "who reads whose data?" — before choosing the mode.
The non-transitive drift. Under plain BACKWARD, v3 is checked only against v2, and v2 only against v1; a field added in v2 and removed in v3 can leave a v1 consumer unable to read v3. Fix: use BACKWARD_TRANSITIVE for long-lived subjects.
Missing defaults. Adding a field without a default and expecting it to be "just an add." Fix: every optional field carries a default; make it a lint rule.

Common interview probes on compatibility.

"What is the default compatibility mode?" — BACKWARD.
"Add a field — is it safe?" — only with a default under BACKWARD.
"Difference between BACKWARD and BACKWARD_TRANSITIVE?" — last version vs all versions.
"Which mode lets producers deploy first?" — FORWARD (or FULL).

Worked example — adding an optional field under BACKWARD

Detailed explanation. The canonical safe evolution: a producer wants to add a loyalty_tier field to the transactions event. Under BACKWARD, this is legal iff the field has a default. Walk through the two schema versions and the resolution.

v1. {id, amount_cents, currency}.
v2. adds loyalty_tier with "default": "none".
Resolution. A v2 consumer reading a v1 record (no loyalty_tier) fills "none".

Question. Show the v1 and v2 Avro schemas and prove the change keeps BACKWARD compatibility.

Input.

Field	v1	v2
id	long	long
amount_cents	long	long
currency	string	string
loyalty_tier	—	string, default "none"

Code.

// transactions-v1.avsc
{
  "type": "record",
  "name": "Transaction",
  "namespace": "payments",
  "fields": [
    {"name": "id",           "type": "long"},
    {"name": "amount_cents", "type": "long"},
    {"name": "currency",     "type": "string"}
  ]
}

// transactions-v2.avsc  — adds loyalty_tier WITH a default
{
  "type": "record",
  "name": "Transaction",
  "namespace": "payments",
  "fields": [
    {"name": "id",           "type": "long"},
    {"name": "amount_cents", "type": "long"},
    {"name": "currency",     "type": "string"},
    {"name": "loyalty_tier", "type": "string", "default": "none"}
  ]
}

# Ask the registry whether v2 is compatible under BACKWARD — no register
curl -s -X POST \
  http://schema-registry:8081/compatibility/subjects/payments.transactions-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data-binary @transactions-v2-wrapped.json
# -> {"is_compatible": true}

Step-by-step explanation.

v1 has three required fields; a v1 producer writes records without any loyalty_tier.
v2 adds loyalty_tier with "default": "none". The default is the load-bearing detail: it is what a new reader substitutes when it decodes an old record that lacks the field.
Under BACKWARD, the registry simulates a v2 (new) reader consuming v1 (old) data. Because the missing field has a default, resolution succeeds — is_compatible: true.
Had we omitted the default, a v2 reader would have no value to supply for loyalty_tier when reading a v1 record; the registry would answer is_compatible: false and the CI gate would fail the PR.
Deployment order: register v2, upgrade consumers to v2 (they can still read the v1 backlog), then upgrade producers to emit v2. No lockstep, no downtime.

Output.

Reader schema	Record written under	Result
v2	v1 (no loyalty_tier)	reads OK; loyalty_tier = "none"
v2	v2	reads OK; loyalty_tier from payload
v1	v1	reads OK
v1	v2	ignores unknown field (also FORWARD-safe here)

Rule of thumb. Every added field carries a default. With a default, an add is BACKWARD-safe and usually FORWARD-safe, which quietly earns you FULL for free. Make "no default" a CI lint failure.

Worked example — the transitive-drift trap

Detailed explanation. Plain BACKWARD checks only the latest version. Over several releases a subject can walk itself into a state where an old consumer that never upgraded can no longer read the newest data — even though every single step was "compatible." Walk through a three-version drift.

v1. {id, amount_cents, note (default "")}.
v2. removes note (legal under BACKWARD because it had a default) — checked only against v1.
v3. re-adds note as a required field with no default — checked only against v2 (which has no note), so it passes.
Break. A consumer still on v1's reader logic that expected note optional now meets a v3 producer that made it required-shaped; a v1-era reader resolving v3 data hits an incompatibility the non-transitive check never evaluated.

Question. Demonstrate why BACKWARD_TRANSITIVE would have caught the v3 change, and set the subject to it.

Input.

Version	Change	Non-transitive check	Transitive check
v2	remove `note` (had default)	vs v1 → pass	vs v1 → pass
v3	add `note` required, no default	vs v2 → pass	vs v1 and v2 → fail

Code.

# Tighten the subject to check against ALL prior versions
curl -X PUT \
  http://schema-registry:8081/config/payments.transactions-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "BACKWARD_TRANSITIVE"}'

# Now the v3 compatibility call is evaluated against v1 AND v2
curl -s -X POST \
  http://schema-registry:8081/compatibility/subjects/payments.transactions-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data-binary @transactions-v3-wrapped.json
# -> {"is_compatible": false, "messages": ["reader incompatible with schema version 1"]}

// The offending v3 field — required, no default = the drift
{"name": "note", "type": "string"}   // add a "default": "" to make it safe

Step-by-step explanation.

Under plain BACKWARD, v2 is validated only against v1 and v3 only against v2. Each hop is individually legal, so the registry accepts the whole chain.
The hazard is a consumer that pinned v1 and never upgraded. Non-transitive checking never asks "can a v1 reader handle v3 data?" — it only ever compared adjacent versions.
Switching the subject to BACKWARD_TRANSITIVE changes the predicate: v3 is now checked against both v1 and v2. The required, default-less note fails against v1, and the registry returns is_compatible: false.
The fix is to give note a default (e.g. ""), restoring compatibility against every prior version.
The lesson: long-lived subjects with slow or pinned consumers should default to the *_TRANSITIVE variant. The extra checks are cheap; a silent v1-consumer break is not.

Output.

Mode	v3 accepted?	v1 consumer safe?
BACKWARD	yes	no (silent break)
BACKWARD_TRANSITIVE	no (until default added)	yes (once fixed)

Rule of thumb. For any subject with consumers that may not upgrade promptly, use the *_TRANSITIVE mode. Non-transitive is fine only when you can guarantee every consumer is at most one version behind.

Worked example — an illegal change the registry rejects

Detailed explanation. Not every change is negotiable. Narrowing a type or removing a required field is breaking under BACKWARD no matter what you do short of a new subject. Walk through a producer that tries to change amount_cents from long to int to "save space," and show the rejection.

v1. amount_cents: long.
v2 attempt. amount_cents: int — a narrowing type change.
Why it breaks. A v2 (int) reader cannot decode a v1 record whose long value exceeds the int range; Avro forbids the narrowing promotion.

Question. Show the rejected change and the correct alternative (a new field or a widening).

Input.

Change	Legal under BACKWARD?	Reason
int → long	yes	widening promotion
long → int	no	narrowing; data loss risk
add field, no default	no	new reader can't fill old data
remove field with default	yes	reader ignores it

Code.

// v2 attempt — narrowing long to int (ILLEGAL)
{"name": "amount_cents", "type": "int"}

curl -s -X POST \
  http://schema-registry:8081/compatibility/subjects/payments.transactions-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data-binary @transactions-v2-narrow.json
# -> {"is_compatible": false,
#     "messages": ["READER_INCOMPATIBLE: type long cannot be read as int"]}

// Correct alternative — keep long, add a NEW optional field if you need a variant
{"name": "amount_cents",       "type": "long"},
{"name": "amount_minor_units", "type": ["null","int"], "default": null}

Step-by-step explanation.

The producer wants int to shave bytes, but amount_cents was written as long; existing records may hold values beyond int's range.
Avro permits only widening promotions (int → long), never narrowing (long → int), because narrowing risks truncating live data. The registry returns is_compatible: false.
The CI gate fails the PR at review time, so the truncating change never reaches a running consumer — exactly the outcome the compatibility mode exists to produce.
The correct move is either to leave the type alone or, if a genuinely different representation is needed, add a new optional field (a nullable union with a default) and migrate consumers deliberately.
If the change is truly unavoidable and breaking (a semantic redefinition), the honest answer is a new subject / new topic and a dual-publish migration — never a silent narrowing.

Output.

Attempt	Registry verdict	Correct path
long → int	rejected	keep long
add required field	rejected	add with default
new optional field	accepted	nullable union + default
new subject for breaking redesign	n/a	dual-publish migration

Rule of thumb. Type changes are legal only when widening. When you truly need a breaking redesign, cut a new subject and dual-publish — never fight the compatibility check by disabling it with NONE.

Senior interview question on compatibility modes

A senior interviewer might ask: "A shared orders topic feeds a fast-moving analytics team and a slow-moving finance system that upgrades twice a year. The analytics producers want to iterate weekly. Which compatibility mode do you set on the subject, what deployment order does it imply, and how do you keep the finance consumer safe over a year of weekly changes?"

Solution Using BACKWARD_TRANSITIVE with default-carrying optional fields

# 1. Set the subject to BACKWARD_TRANSITIVE — every new version is checked
#    against ALL prior versions, protecting the finance consumer that lags.
curl -X PUT http://schema-registry:8081/config/orders-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "BACKWARD_TRANSITIVE"}'

// 2. Every weekly change is an OPTIONAL, default-carrying field add
{
  "type": "record", "name": "Order", "namespace": "sales",
  "fields": [
    {"name": "order_id",     "type": "long"},
    {"name": "total_cents",  "type": "long"},
    {"name": "status",       "type": "string"},
    {"name": "promo_code",   "type": ["null","string"], "default": null},
    {"name": "channel",      "type": "string", "default": "web"}
  ]
}

# 3. CI gate on every schema PR — fail the build if the change is not
#    compatible against every historical version.
curl -s -X POST \
  http://schema-registry:8081/compatibility/subjects/orders-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data-binary @order-next.json | jq -e '.is_compatible == true'

Step-by-step trace.

Concern	Choice	Reasoning
Compatibility mode	BACKWARD_TRANSITIVE	finance consumer lags many versions; check all history
Every change shape	optional field + default	keeps BACKWARD (and usually FORWARD) intact
Deploy order	register → consumers → producers	new-reader-reads-old-data is the safe order
Finance consumer	never forced to redeploy	old reader still decodes new data via defaults
Breaking redesign	new subject + dual-publish	never disable the check

After the policy is set, the analytics team ships weekly additive changes; each PR is validated against the entire version history, so the finance consumer — which may be twenty versions behind — can still decode the newest orders records because every added field is optional with a default. A genuinely breaking redesign is routed to a new subject, never forced through the gate.

Output:

Metric	Value
Compatibility mode	BACKWARD_TRANSITIVE
Weekly changes shipped safely	additive, default-carrying
Finance-consumer forced redeploys	0
Versions the v1 finance reader can read	all
Breaking changes routed to new subject	100%

Why this works — concept by concept:

BACKWARD_TRANSITIVE — checks each new version against every prior version, not just the last, which is exactly what protects a consumer that is many versions behind.
Optional field with default — the atomic safe change: a new reader fills the default for old data, so the add never breaks a lagging consumer; it also stays FORWARD-safe, so producers can lead.
Register-then-consumers-then-producers order — BACKWARD's guarantee ("new reader reads old data") is exactly the property that lets consumers move ahead of producers without a coordinated cutover.
New subject for breaking redesigns — the escape hatch that keeps the gate honest; you never reach for NONE, you reach for a new contract and a dual-publish window.
Cost — a handful of extra compatibility comparisons per register (against N historical versions) and the discipline of always adding defaults. The eliminated cost is the twice-a-year finance-system break. O(N-versions) per compatibility check, which is negligible.

SQL
Topic — data-validation
Data-validation and contract-check problems

Practice →

SQL Topic — streaming Streaming schema-evolution problems

Practice →

3. Confluent Schema Registry deep dive

`Confluent Schema Registry` is the reference implementation — schemas live in a compacted `_schemas` topic, the wire carries a magic byte plus a 4-byte schema ID, and subject naming decides scope

The mental model in one line: the Confluent Schema Registry is a REST service whose source of truth is a compacted Kafka topic called _schemas, which serves schemas by subject and by global ID, and whose client serializers define the de-facto wire format — a single magic byte (0x0), a 4-byte big-endian schema ID, then the Avro/Protobuf/JSON-Schema payload — so a consumer reads the ID off the front of each record and fetches the exact writer schema to deserialize against. Everything else — subject naming, compatibility scope, the REST endpoints — is layered on this core.

The wire format — five bytes then payload.

Byte 0 — magic byte. Always 0x0. Signals the Confluent framing; a consumer that reads a different first byte knows this is not a registry-framed record.
Bytes 1–4 — schema ID. A 4-byte big-endian integer: the global schema ID assigned by the registry when the schema was registered. Not the subject version — the global ID.
Bytes 5…N — payload. The serialized Avro (or Protobuf/JSON Schema) body, written without the embedded schema, because the ID already points at it.
Consumer flow. Read byte 0 (assert magic), read bytes 1–4 (schema ID), GET /schemas/ids/{id} (cached), deserialize the rest with that writer schema resolved against the consumer's reader schema.

The _schemas topic — the source of truth.

What it is. A single-partition, cleanup.policy=compact Kafka topic. Every register/config/delete is an event; the registry is a materialized view over it.
Why compacted. Compaction retains the latest value per key indefinitely, so the full schema history survives log cleaning.
HA model. Multiple registry nodes; one is the leader (elected) and handles writes to _schemas; followers serve reads. Losing the registry does not stop existing producers/consumers (they cache), but blocks new schema registrations.

Subjects, versions, and compatibility scope.

Subject. The unit of compatibility. A subject has an ordered list of versions; a compatibility mode is set globally and can be overridden per subject.
Version vs global ID. A subject version (1, 2, 3…) is local to the subject; the global schema ID is unique across the whole registry and is what goes on the wire. The same schema registered under two subjects shares one global ID.
Key endpoints. POST /subjects/{s}/versions (register), GET /subjects/{s}/versions/{v}, POST /compatibility/subjects/{s}/versions/{v} (check without registering), PUT /config/{s} (set per-subject compatibility).

Subject-naming strategies — the multi-event lever.

TopicNameStrategy (default). Subject = {topic}-key / {topic}-value. One schema per topic per key/value. Simple; a topic carries a single record type.
RecordNameStrategy. Subject = the record's fully-qualified name (payments.Transaction). Lets one topic carry many record types, each independently versioned. Compatibility is scoped per record type, not per topic.
TopicRecordNameStrategy. Subject = {topic}-{record-fqn}. Multiple record types per topic and the same record type can evolve differently on different topics.

Common interview probes on Confluent.

"What is on the wire?" — magic byte + 4-byte global schema ID + payload.
"Where does the registry store schemas?" — a compacted _schemas Kafka topic.
"How do you put multiple event types on one topic?" — RecordNameStrategy or TopicRecordNameStrategy.
"Version vs schema ID?" — version is per-subject; global ID is registry-wide and goes on the wire.

Worked example — register and evolve a subject via REST

Detailed explanation. The canonical Confluent flow with no client library: register v1, add an optional field for v2, and read back the subject's versions — all via curl. Avro schemas are sent as a JSON-escaped string inside a {"schema": "..."} envelope. Walk through it.

Register. POST /subjects/orders-value/versions with the escaped schema.
Evolve. POST v2 (adds a defaulted field) to the same subject.
Inspect. GET /subjects/orders-value/versions lists [1, 2].

Question. Register a v1 schema, evolve to v2, and fetch the version list and the global IDs.

Input.

Call	Endpoint	Effect
register v1	POST /subjects/orders-value/versions	returns {"id": N}
register v2	POST /subjects/orders-value/versions	returns {"id": M}
list versions	GET /subjects/orders-value/versions	[1, 2]
fetch by id	GET /schemas/ids/N	the v1 schema

Code.

# 1. Register v1 — note the schema is a JSON string inside "schema"
curl -s -X POST http://schema-registry:8081/subjects/orders-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{\"type\":\"record\",\"name\":\"Order\",\"namespace\":\"sales\",\"fields\":[{\"name\":\"order_id\",\"type\":\"long\"},{\"name\":\"total_cents\",\"type\":\"long\"}]}"}'
# -> {"id": 101}

# 2. Register v2 — adds channel with a default (BACKWARD-safe)
curl -s -X POST http://schema-registry:8081/subjects/orders-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{\"type\":\"record\",\"name\":\"Order\",\"namespace\":\"sales\",\"fields\":[{\"name\":\"order_id\",\"type\":\"long\"},{\"name\":\"total_cents\",\"type\":\"long\"},{\"name\":\"channel\",\"type\":\"string\",\"default\":\"web\"}]}"}'
# -> {"id": 102}

# 3. List the subject's versions
curl -s http://schema-registry:8081/subjects/orders-value/versions
# -> [1, 2]

# 4. Fetch the global schema by ID (what a consumer does off the wire)
curl -s http://schema-registry:8081/schemas/ids/101

Step-by-step explanation.

The register call wraps the Avro schema as an escaped JSON string in the schema field. The registry parses it, checks compatibility against the subject's latest version (here there is none yet, so v1 registers freely), and returns a global ID 101.
The v2 call adds channel with "default":"web". The registry checks it against v1 under the subject's mode (BACKWARD by default), finds it compatible, and assigns a new global ID 102 and subject version 2.
GET /subjects/orders-value/versions returns [1, 2] — the ordered version list local to this subject.
GET /schemas/ids/101 returns the schema for global ID 101 — this is exactly the call a consumer makes after reading the 4-byte ID off the wire. The result is cached client-side so the lookup happens once per unseen ID.
The distinction to state in an interview: versions are [1,2] (subject-local), while the wire carried 101/102 (registry-global). Same schema under a second subject would reuse the global ID.

Output.

Call	Result
register v1	{"id": 101}
register v2	{"id": 102}
GET versions	[1, 2]
GET /schemas/ids/101	the v1 Order schema

Rule of thumb. In dev, let the serializer auto-register; in prod, register via CI with auto.register.schemas=false so schemas enter the registry only through the reviewed, compatibility-gated pipeline.

Worked example — serializer config and the wire bytes

Detailed explanation. A producer using KafkaAvroSerializer never sees the schema ID directly, but it is worth being able to describe the exact bytes. Walk through the producer config and a byte-level view of one serialized record.

Config. value.serializer=KafkaAvroSerializer, schema.registry.url=....
On register. The serializer registers (dev) or looks up (prod) the schema, caches the ID.
On the wire. 0x00 + 00 00 00 65 (ID 101) + Avro body.

Question. Configure the producer and show the leading bytes of a serialized Order record.

Input.

Setting	Value
value.serializer	io.confluent.kafka.serializers.KafkaAvroSerializer
schema.registry.url	http://schema-registry:8081
auto.register.schemas	false (prod)
use.latest.version	true

Code.

// Producer config (Java) — Avro value serialization via the registry
Properties props = new Properties();
props.put("bootstrap.servers", "kafka:9092");
props.put("key.serializer",
    "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer",
    "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://schema-registry:8081");
props.put("auto.register.schemas", false);   // prod: CI registers
props.put("use.latest.version", true);

var producer = new KafkaProducer<String, GenericRecord>(props);
producer.send(new ProducerRecord<>("orders", order.get("order_id").toString(), order));

# Byte layout of one serialized Order record (schema global ID 101 = 0x65)
+------+-------------+-------------------------------+
| 0x00 | 00 00 00 65 |  <avro-encoded order body>    |
+------+-------------+-------------------------------+
  magic   schema ID          payload (no schema)
  (1 B)   (4 B, BE int)

Step-by-step explanation.

KafkaAvroSerializer is wired as the value serializer with the registry URL; the key here is a plain string.
On the first send of a given schema, the serializer resolves the schema ID — in prod it looks up the ID (auto.register.schemas=false), because registration happened earlier via CI; in dev it would register on the fly.
It writes the magic byte 0x00, then the 4-byte big-endian schema ID (101 = 0x00000065), then the Avro-encoded body with no embedded schema.
A consumer reverses this: read 0x00, read 0x00000065, GET /schemas/ids/101 (cached), deserialize the remaining bytes.
use.latest.version=true tells the serializer to use the latest registered schema for the subject rather than the local writer schema, which is the safe production default when CI owns registration.

Output.

Byte range	Meaning	Example
[0]	magic byte	0x00
[1..4]	global schema ID	0x00000065 (101)
[5..]	Avro payload	binary body

Rule of thumb. The wire carries the global schema ID, not the subject version and not the schema. When debugging "why can't this consumer deserialize?", read the first five bytes: a wrong magic byte or an unknown ID is the tell.

Worked example — RecordNameStrategy for multi-event topics

Detailed explanation. Sometimes one topic must carry several related event types in order — OrderPlaced, OrderShipped, OrderCancelled on a single order-events topic to preserve per-order ordering. TopicNameStrategy cannot do this (one schema per topic-value). RecordNameStrategy keys the subject on the record's fully-qualified name instead. Walk through the config.

Problem. Three event types must share a topic for ordering.
Strategy. RecordNameStrategy → subjects sales.OrderPlaced, sales.OrderShipped, sales.OrderCancelled.
Compatibility. Scoped per record type, independently versioned.

Question. Configure the producer to place three record types on one topic with independent subjects.

Input.

Setting	Value
value.subject.name.strategy	io.confluent.kafka.serializers.subject.RecordNameStrategy
topic	order-events
record types	OrderPlaced, OrderShipped, OrderCancelled
subjects	sales.OrderPlaced, sales.OrderShipped, sales.OrderCancelled

Code.

// Producer: many event types, one topic, subject = record FQN
props.put("value.serializer",
    "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("value.subject.name.strategy",
    "io.confluent.kafka.serializers.subject.RecordNameStrategy");

// All three go to the SAME topic, keyed by order_id for ordering
producer.send(new ProducerRecord<>("order-events", orderId, orderPlaced));
producer.send(new ProducerRecord<>("order-events", orderId, orderShipped));
producer.send(new ProducerRecord<>("order-events", orderId, orderCancelled));

# The registry now shows three record-named subjects, not one topic subject
curl -s http://schema-registry:8081/subjects
# -> ["sales.OrderPlaced","sales.OrderShipped","sales.OrderCancelled"]

Step-by-step explanation.

The default TopicNameStrategy would demand a single order-events-value schema, forbidding three distinct record types on the topic.
Setting value.subject.name.strategy=RecordNameStrategy makes the subject the record's fully-qualified name — sales.OrderPlaced, etc. — so each type registers and evolves independently.
All three events go to the same topic with the same order_id key, so Kafka keeps them in per-order order on one partition — the reason for co-locating them.
Compatibility is now scoped per record type: evolving OrderShipped cannot break OrderPlaced, because they are different subjects.
The trade-off: consumers must handle a heterogeneous topic (dispatch on record type). TopicRecordNameStrategy adds the topic prefix if the same record type must evolve differently across topics.

Output.

Strategy	Subject(s) for order-events	Multi-type topic?
TopicNameStrategy	order-events-value	no
RecordNameStrategy	sales.OrderPlaced / Shipped / Cancelled	yes
TopicRecordNameStrategy	order-events-sales.OrderPlaced …	yes, per-topic scoped

Rule of thumb. Reach for RecordNameStrategy only when several event types genuinely must share a topic for ordering; otherwise keep TopicNameStrategy for its one-schema-per-topic simplicity. Consumers of a multi-type topic must dispatch on record type.

Senior interview question on Confluent internals

A senior interviewer might ask: "Walk me through what happens, byte by byte, when a producer sends an Avro record through the Confluent Schema Registry and a consumer reads it back. Cover registration, the wire format, the _schemas topic, and what breaks if the registry is down."

Solution Using the KafkaAvroSerializer end-to-end with an ID cache

1. PRODUCER SERIALIZE
   - value.serializer = KafkaAvroSerializer
   - resolve schema id (prod: lookup, dev: auto-register) -> id = 101
   - write: [0x00][00 00 00 65][avro body]     <- 5-byte header + payload
   - registry stores the schema in the compacted _schemas topic

2. CONSUMER DESERIALIZE
   - read [0]  -> assert magic byte 0x00
   - read [1..4] -> schema id 101
   - GET /schemas/ids/101 (cached after first fetch)
   - resolve writer schema (id 101) against reader schema -> POJO/GenericRecord

# Prove the header on a raw fetch (first 5 bytes = magic + id)
kafka-console-consumer --bootstrap-server kafka:9092 \
  --topic orders --from-beginning --max-messages 1 \
  --property print.value=true \
  --value-deserializer org.apache.kafka.common.serialization.ByteArrayDeserializer \
  | xxd | head -1
# 0000: 0000 0000 65..   <- 0x00 magic, 0x00000065 = id 101

Step-by-step trace.

Stage	Action	Detail
Register	serializer resolves id	101, stored in `_schemas`
Serialize	write header + body	`0x00` + `00 00 00 65` + Avro
Produce	send to topic	key=order_id, value=framed bytes
Consume header	read magic + id	assert `0x00`; id = 101
Fetch schema	GET /schemas/ids/101	cached after first call
Deserialize	resolve writer vs reader	GenericRecord / POJO

If the registry is unreachable, producers and consumers that have already cached the schema IDs they use keep working; only new schema registrations (a producer emitting a never-seen schema) and cold consumers (that have not cached an ID they encounter) block. This is why the _schemas topic is compacted and the registry runs multiple nodes — the contract must survive a node loss.

Output:

Byte	Value	Meaning
0	0x00	magic byte
1–4	00 00 00 65	global schema ID 101
5…N	binary	Avro payload (no embedded schema)
lookup	GET /schemas/ids/101	writer schema, cached
result	GenericRecord	resolved against reader schema

Why this works — concept by concept:

Magic byte plus 4-byte ID — five bytes of framing turn every record into a self-describing pointer at a registered schema, so the payload stays compact and the consumer always knows which writer schema to resolve.
Compacted _schemas topic — the registry is a materialized view over an append-only, log-compacted Kafka topic, so schema history is durable and survives log cleaning and node restarts.
Client-side ID cache — the consumer fetches each schema ID once and caches it, so the registry is on the cold path only; steady-state throughput does not depend on a registry round-trip per message.
Leader-based HA — multiple registry nodes with a single write leader mean a node loss degrades to read-only (registrations blocked) rather than a full outage of the running pipeline.
Cost — five bytes per message on the wire and one cached REST lookup per unseen schema ID. The eliminated cost is shipping the full schema with every record and the coordination of out-of-band schema sharing. O(1) per message; O(distinct-schemas) lookups.

SQL
Topic — streaming
Streaming serialization and wire-format problems

Practice →

SQL Topic — json JSON and schema-encoding problems

Practice →

4. Apicurio and AWS Glue Schema Registry

`Apicurio` is the open-source, Confluent-compatible registry and `AWS Glue Schema Registry` is the AWS-native, IAM-governed one — both speak the same producer contract if you keep it Avro-first

The mental model in one line: Apicurio Registry stores schemas as artifacts inside groups and exposes both a native API and a Confluent-compatible ccompat API — so existing Confluent serializers work against it by changing only the URL — while AWS Glue Schema Registry nests schemas under registry → schema → version, governs access with IAM instead of ACLs, and ships its own serializers (GlueSchemaRegistryKafkaSerializer) with a UUID-based wire header rather than Confluent's 4-byte int — and the practical decision between them is driven by cloud lock-in, licensing, and whether you need multi-format artifacts beyond Avro. Both are legitimate; the wrong reason to pick one is "it's the one I've heard of."

Apicurio Registry — the open alternative.

Model. Artifacts (a schema/contract) live inside groups (a namespace); each artifact has ordered versions. Supports Avro, Protobuf, JSON Schema, OpenAPI, AsyncAPI, GraphQL, WSDL, XSD.
Two APIs. A native Apicurio REST API and a Confluent-compatible ccompat API (e.g. /apis/ccompat/v7). The ccompat endpoint mimics Confluent's routes so KafkaAvroSerializer works unchanged.
Storage. Pluggable — in-memory (dev), SQL (Postgres), or a Kafka topic (like Confluent). Apache-2.0 licensed, self-hostable anywhere.
Content rules. Per-artifact or global rules — VALIDITY (is the content well-formed) and COMPATIBILITY (BACKWARD/FORWARD/FULL/NONE) — the Apicurio equivalent of Confluent's compatibility config.

AWS Glue Schema Registry — the AWS-native option.

Model. registry (a namespace) → schema (named contract) → schema version. data-format (AVRO/PROTOBUF/JSON) and compatibility are set on the schema.
Serializers. AWS's own GlueSchemaRegistryKafkaSerializer / Avro serializer; native to MSK, Kinesis Data Streams, Kinesis Data Analytics/Flink, and Lambda.
Governance. IAM policies gate glue:RegisterSchemaVersion, glue:GetSchemaVersion, etc. No separate ACL system — schema access is part of your existing AWS IAM posture.
Wire format. A header byte plus an 8-byte compression indicator and a 16-byte UUID schema version id — not Confluent's 5-byte framing. So a Glue move is a serializer swap, not just a URL change.

Interop and migration.

Confluent → Apicurio. Point the serializer at Apicurio's ccompat URL. The wire format is identical (magic byte + 4-byte ID), so producers and consumers do not change code — only configuration.
Confluent/Apicurio → Glue. Requires swapping to Glue serializers because the wire header differs; plan a dual-publish or a topic cut, not a config flip.
Format portability. Keeping schemas Avro-first maximises portability; all three registries treat Avro as a first-class citizen.

Common interview probes on alternatives.

"How is Apicurio different from Confluent?" — open-source, group/artifact model, multi-format, and a Confluent-compatible API.
"What is ccompat?" — Apicurio's Confluent-compatible REST endpoint that lets Confluent clients work unchanged.
"How does Glue govern access?" — IAM, not a bespoke ACL system.
"Can I move Confluent → Glue by changing a URL?" — no; the wire header differs, so serializers change.

Worked example — swapping Confluent for Apicurio via ccompat

Detailed explanation. A team on self-managed Kafka wants to drop Confluent's registry and self-host Apicurio without touching producer/consumer code. Because Apicurio exposes a Confluent-compatible API, the change is a single URL. Walk through it.

Before. schema.registry.url=http://confluent-sr:8081.
After. schema.registry.url=http://apicurio:8080/apis/ccompat/v7.
Code changes. None — same KafkaAvroSerializer, same wire format.

Question. Repoint an existing Confluent-serializer producer at Apicurio and register a schema through the ccompat API.

Input.

Setting	Confluent	Apicurio (ccompat)
serializer	KafkaAvroSerializer	KafkaAvroSerializer (unchanged)
registry URL	http://confluent-sr:8081	http://apicurio:8080/apis/ccompat/v7
register route	POST /subjects/{s}/versions	POST /subjects/{s}/versions (ccompat)
wire format	magic byte + 4-byte ID	identical

Code.

# Before — Confluent
schema.registry.url=http://confluent-sr:8081
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer

# After — Apicurio via the Confluent-compatible ccompat API
schema.registry.url=http://apicurio:8080/apis/ccompat/v7
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer

# Register through Apicurio's ccompat endpoint — same shape as Confluent
curl -s -X POST \
  http://apicurio:8080/apis/ccompat/v7/subjects/orders-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{\"type\":\"record\",\"name\":\"Order\",\"namespace\":\"sales\",\"fields\":[{\"name\":\"order_id\",\"type\":\"long\"}]}"}'
# -> {"id": 1}

# Set a COMPATIBILITY rule (Apicurio's equivalent of Confluent config)
curl -s -X PUT http://apicurio:8080/apis/ccompat/v7/config/orders-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "BACKWARD"}'

Step-by-step explanation.

The producer keeps KafkaAvroSerializer verbatim; only schema.registry.url changes to Apicurio's ccompat base path. The serializer neither knows nor cares that a different server answers.
Registration uses the same Confluent route shape (/subjects/{s}/versions) because ccompat re-implements it; Apicurio internally maps the subject onto its group/artifact model.
The wire format is byte-identical — magic byte plus 4-byte ID — so records produced against Apicurio deserialize on any Confluent-serializer consumer and vice versa. This is what makes the swap non-breaking.
Compatibility is configured via the ccompat /config route; under the hood Apicurio stores it as a COMPATIBILITY rule on the artifact.
The migration playbook: stand up Apicurio, replay/import existing schemas, flip the URL in a rolling deploy. No re-serialisation, no consumer coordination.

Output.

Aspect	Result after swap
Producer/consumer code	unchanged
Config change	registry URL only
Wire format	identical (magic + 4-byte ID)
Compatibility	set via ccompat /config
Migration risk	low (rolling URL flip)

Rule of thumb. If portability off Confluent matters, keep the Avro + Confluent-serializer contract; then Apicurio is a URL change via ccompat. Reserve the native Apicurio API for multi-format artifacts (OpenAPI/AsyncAPI) that Confluent does not serve.

Worked example — Glue serializer config with IAM governance

Detailed explanation. An AWS MSK shop wants schemas managed by Glue and access governed by IAM. The producer uses AWS's Glue serializer; the schema's compatibility and data-format are set on the Glue schema; permissions come from an IAM policy. Walk through it.

Registry. A Glue registry payments.
Serializer. GlueSchemaRegistryKafkaSerializer with auto-registration and compatibility on the registry.
IAM. Policy granting glue:GetSchemaVersion / glue:RegisterSchemaVersion on the registry ARN.

Question. Configure a Glue-backed Kafka producer and the IAM policy that governs schema access.

Input.

Setting	Value
registry name	payments
data format	AVRO
compatibility	BACKWARD
serializer	GlueSchemaRegistryKafkaSerializer
auth	IAM (task role / instance role)

Code.

// Glue-backed producer config
Properties props = new Properties();
props.put("bootstrap.servers", "b-1.msk.amazonaws.com:9098");
props.put("value.serializer",
    "com.amazonaws.services.schemaregistry.serializers.GlueSchemaRegistryKafkaSerializer");
props.put("registry.name", "payments");
props.put("schemaAutoRegistrationEnabled", "false");   // register via CI
props.put("compatibility", "BACKWARD");
props.put("dataFormat", "AVRO");
props.put("region", "us-east-1");

// IAM policy — least-privilege schema access on the payments registry
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueSchemaProducer",
      "Effect": "Allow",
      "Action": [
        "glue:GetSchemaVersion",
        "glue:GetSchemaByDefinition",
        "glue:RegisterSchemaVersion",
        "glue:PutSchemaVersionMetadata"
      ],
      "Resource": [
        "arn:aws:glue:us-east-1:111122223333:registry/payments",
        "arn:aws:glue:us-east-1:111122223333:schema/payments/*"
      ]
    }
  ]
}

Step-by-step explanation.

The producer wires AWS's GlueSchemaRegistryKafkaSerializer and names the registry.name, dataFormat, and compatibility. Unlike Confluent, there is no separate registry URL — the SDK resolves the Glue endpoint from the AWS region and credentials.
schemaAutoRegistrationEnabled=false mirrors the Confluent prod pattern: schemas are registered through CI, not silently by the first producer.
Authentication is IAM — the task/instance role's policy decides whether the producer may RegisterSchemaVersion or only GetSchemaVersion. There is no bespoke ACL layer to run.
The IAM policy scopes actions to the payments registry ARN and its schemas, so a producer can be granted read-only (consumer) or read-write (producer) purely through IAM — the same posture that governs the rest of the AWS estate.
The catch to state: Glue's wire header (UUID version id) differs from Confluent's 4-byte int, so this is not a drop-in for existing Confluent-serialized topics — a Glue move is a deliberate serializer migration.

Output.

Aspect	Glue behaviour
Endpoint resolution	region + credentials (no URL)
Access control	IAM policy on registry/schema ARNs
Registration	CI-driven (auto-register off)
Wire header	UUID schema-version-id (not 4-byte int)
Native integrations	MSK, Kinesis, Flink, Lambda

Rule of thumb. Pick Glue when you are AWS-native and want schema access folded into IAM. Accept that its wire format is Glue-specific — a move from Confluent/Apicurio to Glue is a serializer swap and a dual-publish migration, not a URL change.

Worked example — the three-registry decision under real constraints

Detailed explanation. Interviewers love a "which would you pick and why" that hinges on constraints, not preferences. Walk three constraint sets to the registry each implies, and name the disqualifier for the other two each time.

Set A. On-prem, no cloud, wants Apache-2.0 licensing and OpenAPI artifacts too.
Set B. All-in AWS MSK, small team, wants zero extra infra to operate.
Set C. Large multi-team platform on self-managed Kafka, richest ecosystem/tooling matters most.

Question. Map each constraint set to a registry and state why the other two lose.

Input.

Constraint set	Decisive factor
A: on-prem, OSS, multi-format	open-source + multi-format
B: AWS MSK, minimal ops	managed + IAM-native
C: big platform, best ecosystem	maturity + connector/tooling breadth

Code.

Set A -> Apicurio
  Confluent loses: license + no OpenAPI/AsyncAPI artifacts
  Glue loses:      AWS-only, they are on-prem

Set B -> AWS Glue Schema Registry
  Confluent loses: another service to run; not IAM-native
  Apicurio loses:  self-hosting is extra ops the small team does not want

Set C -> Confluent Schema Registry
  Apicurio loses:  ecosystem/tooling breadth is thinner
  Glue loses:      they are self-managed, not AWS; Glue is AWS-coupled

Step-by-step explanation.

Set A's decisive factor is licensing plus multi-format artifacts; Apicurio is Apache-2.0 and serves OpenAPI/AsyncAPI alongside Avro, so it wins while Confluent (license, Avro/Protobuf/JSON only) and Glue (AWS-only) are disqualified.
Set B optimises for not operating a registry; Glue is fully managed and IAM-native, so it removes a service and folds access control into existing AWS posture. Confluent and self-hosted Apicurio both add ops the small team wants to avoid.
Set C optimises for ecosystem maturity at platform scale; Confluent's tooling, connector, and client breadth is the deepest, and the team is self-managed (so Glue's AWS coupling is a liability, and Apicurio's ecosystem is thinner).
Note that all three answers keep Avro as the format, preserving portability if a constraint changes later.
The interview signal is naming the disqualifier for the losers, not just the winner — that shows you reasoned from constraints, not familiarity.

Output.

Constraint set	Registry	Why the other two lose
A: on-prem, OSS, multi-format	Apicurio	Confluent license/format; Glue AWS-only
B: AWS MSK, minimal ops	Glue	Confluent/Apicurio add ops
C: big platform, best ecosystem	Confluent	Apicurio thinner tooling; Glue AWS-coupled

Rule of thumb. Decide from constraints — cloud, licensing, ops appetite, multi-format need — and name why the losers lose. Keep the schema Avro-first so the decision stays reversible.

Senior interview question on registry choice and interop

A senior interviewer might ask: "Your company runs self-managed Kafka with Confluent Schema Registry today, but a mandate says no proprietary licenses on the new platform, and a separate business unit is going all-in on AWS MSK. Design a registry strategy that satisfies both, preserves the existing producers/consumers, and explains what code changes each move requires."

Solution Using Apicurio ccompat for the OSS mandate and Glue for the AWS unit

Strategy
========
1. Self-managed platform (OSS mandate):
     Confluent SR  ->  Apicurio (ccompat API)
     - Change only schema.registry.url to .../apis/ccompat/v7
     - Wire format identical (magic + 4-byte id) -> NO code, NO re-serialisation
     - Import existing schemas into Apicurio; roll the URL out gradually

2. AWS MSK business unit:
     New topics on Glue Schema Registry
     - Swap to GlueSchemaRegistryKafkaSerializer (different wire header)
     - Govern access via IAM; register via CI
     - This IS a serializer change -> dual-publish during any migration

# Self-managed platform: one-line change per app
# schema.registry.url=http://confluent-sr:8081
schema.registry.url=http://apicurio:8080/apis/ccompat/v7
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer

// AWS unit: Glue serializer (new wire header -> not a drop-in)
props.put("value.serializer",
  "com.amazonaws.services.schemaregistry.serializers.GlueSchemaRegistryKafkaSerializer");
props.put("registry.name", "payments");
props.put("dataFormat", "AVRO");
props.put("region", "us-east-1");

Step-by-step trace.

Unit	Move	Code change	Wire change
Self-managed	Confluent → Apicurio ccompat	none	none (identical)
Self-managed	compatibility rules	ccompat /config	n/a
AWS MSK	new topics on Glue	serializer swap	UUID header
AWS MSK	access control	IAM policies	n/a
Both	keep Avro	—	portable format

The OSS mandate is satisfied by moving the self-managed platform to Apicurio through the ccompat API — a URL change per app, no re-serialisation, because the wire format is identical. The AWS business unit adopts Glue for its new MSK topics, accepting a serializer swap and a dual-publish window for any topic that must move, and governs schema access through IAM. Keeping every schema Avro-first means a later consolidation onto one registry is a config-and-serializer exercise, not a data rewrite.

Output:

Requirement	Solution	Cost
No proprietary license	Apicurio via ccompat	URL change only
Preserve producers/consumers	identical wire format	zero code
AWS MSK unit	Glue + IAM	serializer swap
Cross-unit portability	Avro-first everywhere	reversible
Breaking migration (if any)	dual-publish window	temporary duplication

Why this works — concept by concept:

Apicurio ccompat API — Apicurio re-implements Confluent's REST routes and wire format, so an existing Confluent-serializer app moves with a URL change and no re-serialisation, satisfying the OSS mandate cheaply.
Glue plus IAM — the AWS unit folds schema access into its existing IAM posture and removes a service to operate, at the price of a Glue-specific wire header that makes the move a deliberate serializer swap.
Avro-first everywhere — standardising on Avro keeps the contract portable across all three registries, so a future consolidation is config, not a data rewrite.
Dual-publish for breaking moves — the only move with a different wire format (to Glue) is handled with a dual-publish window rather than a risky flip, preserving consumers throughout.
Cost — one URL change per self-managed app, a serializer swap plus IAM policies for the AWS unit, and a temporary dual-publish for any Glue migration. The eliminated cost is a proprietary-license bill and a bespoke ACL system. O(apps) config changes; no per-message cost change.

SQL
Topic — streaming
Streaming registry and interop problems

Practice →

SQL Topic — design Design problems on platform and registry choice

Practice →

5. Governance, ownership, and CI enforcement

`governance` is what turns a registry from storage into a control plane — CI compatibility gates, clear ownership, and access control stop a breaking change before it reaches a running consumer

The mental model in one line: schema governance is the set of controls that decide who may register what and when a change is allowed to reach production — the producing team owns each subject, a CI compatibility check gates every schema change at pull-request time (not at register time in prod), schemas live in git as code, naming and validity rules are linted, and access is scoped by RBAC or IAM — so that a breaking change fails a red build rather than a running consumer. The registry can reject a bad schema at register time, but the cheap, humane place to catch it is CI, before the producer ever calls register.

Ownership — the organisational half.

Producer owns the subject. The team that writes to a topic owns its value schema and its compatibility policy. Consumers subscribe to the published contract; they do not get to mutate it.
A subject has exactly one owner. Ambiguous ownership is how the "wiki-documented schema" rots. Record the owner in metadata (Confluent schema metadata, Apicurio artifact labels, Glue tags).
Consumers register interest. Consumers should be discoverable (who reads this subject?) so a proposed change can be socialised with the people it affects.

The CI compatibility gate — shift the check left.

Check, don't register. Use POST /compatibility/subjects/{s}/versions/latest (Confluent/Apicurio ccompat) or the Maven/Gradle schema-registry plugin's test-compatibility goal to answer "would this register?" without registering.
Fail the PR. If is_compatible: false, fail the build. The breaking change never merges, so it never reaches the register step in prod.
Register on merge. Only after the PR merges does the pipeline actually register the new version — auto.register.schemas=false in prod guarantees producers cannot sneak a schema in at runtime.

Rules, linting, and lifecycle.

Validity rules. Reject malformed schemas (Apicurio VALIDITY, or a schema-lint step) — no field without a type, no missing namespace.
Naming/lint conventions. Enforce subject naming ({topic}-value), field naming (snake_case), mandatory doc strings, and "every optional field has a default."
Deprecation and retirement. Mark fields deprecated in the doc before removal; give consumers a window; only then remove (with a default already present).
Schema as code. Schemas live in a git repo; the registry is a deployment target, not the editing surface. This gives review, history, and rollback.

Access control.

Confluent. RBAC / ACLs on subjects — who may read, write, or change config.
Apicurio. Role-based access plus per-artifact/global rules.
Glue. IAM policies on registry/schema ARNs — RegisterSchemaVersion vs GetSchemaVersion.

Common interview probes on governance.

"How do you stop a producer shipping a breaking change?" — CI compatibility check pre-merge; auto.register.schemas=false in prod.
"Who owns a schema?" — the producing team; one accountable owner.
"Where do schemas live?" — in git, as code; the registry is a deployment target.
"How do you retire a field?" — deprecate in doc, give a window, remove with a default present.

Worked example — a CI compatibility gate in the pipeline

Detailed explanation. The canonical governance control: a CI job that, on every schema PR, checks the proposed schema against the registry's latest version and fails the build if it is incompatible. No registration happens in CI for a PR — only the check. Walk through a GitHub-Actions-style job.

Trigger. PR touching schemas/**.
Check. POST /compatibility/.../versions/latest; assert is_compatible == true.
Register. Only on merge to main, a separate job registers the version.

Question. Write the CI job that gates schema PRs on compatibility and the merge job that registers.

Input.

Stage	Trigger	Action
PR check	pull_request on schemas/**	compatibility check, fail if false
Merge register	push to main	register the new version
Prod producers	runtime	auto.register.schemas=false

Code.

# .github/workflows/schema-gate.yml
name: schema-compatibility-gate
on:
  pull_request:
    paths: ["schemas/**"]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Check compatibility (does NOT register)
        run: |
          for f in schemas/*-value.avsc; do
            subject="$(basename "$f" .avsc)"
            payload=$(jq -Rs '{schema: .}' < "$f")
            resp=$(curl -s -X POST \
              "$SR_URL/compatibility/subjects/$subject/versions/latest" \
              -H "Content-Type: application/vnd.schemaregistry.v1+json" \
              -d "$payload")
            echo "$subject -> $resp"
            echo "$resp" | jq -e '.is_compatible == true' > /dev/null \
              || { echo "BREAKING change in $subject"; exit 1; }
          done
        env:
          SR_URL: ${{ secrets.SCHEMA_REGISTRY_URL }}

# Merge job (push to main) — NOW register the reviewed version
for f in schemas/*-value.avsc; do
  subject="$(basename "$f" .avsc)"
  jq -Rs '{schema: .}' < "$f" | curl -s -X POST \
    "$SR_URL/subjects/$subject/versions" \
    -H "Content-Type: application/vnd.schemaregistry.v1+json" -d @-
done

Step-by-step explanation.

The PR job runs only when files under schemas/** change. For each *-value.avsc, it derives the subject from the filename and wraps the schema in the {"schema": "..."} envelope with jq.
It calls /compatibility/.../versions/latest, which asks the registry "would this be compatible?" without registering. This is the crucial distinction: the PR check has no side effect on the registry.
If is_compatible is not true, the step exits non-zero and the build fails, blocking the merge. The author sees the break at review time, in their own PR.
Only the merge job (on push to main) actually registers the new version — after human review has approved it. This keeps the registry's history clean of speculative PR schemas.
Prod producers run with auto.register.schemas=false, so even a misconfigured app cannot register a schema at runtime; the pipeline is the only path in. The gate is therefore complete: no un-reviewed, un-checked schema can ever reach the registry.

Output.

Event	CI result	Registry effect
PR adds field with default	check passes	none (register on merge)
PR removes required field	check fails, PR blocked	none
Merge to main	register job runs	new version registered
Runtime producer	n/a	cannot auto-register

Rule of thumb. Check compatibility on the PR, register on merge, and set auto.register.schemas=false in prod. These three together make a breaking change a red build instead of a 3 AM page.

Worked example — schema-lint rules beyond compatibility

Detailed explanation. Compatibility is necessary but not sufficient; a schema can be perfectly compatible yet violate house style (missing docs, a field with no default, a bad namespace). A lint step enforces the conventions the registry does not. Walk through a small linter.

Rules. Namespace present; every field has a doc; every non-key field has a default; snake_case names.
When. Same PR job, before the compatibility check.
Effect. Fails the build on a style violation with a precise message.

Question. Write a lint that rejects a schema missing defaults or docs, and show it catching a bad field.

Input.

Rule	Pass	Fail
namespace present	"namespace": "sales"	missing
field has doc	"doc": "..."	absent
optional field has default	"default": null	absent
snake_case	order_id	orderId

Code.

# schema_lint.py — house rules the registry does not enforce
import json, re, sys

def lint(path: str) -> list[str]:
    s = json.load(open(path))
    errs: list[str] = []
    if not s.get("namespace"):
        errs.append("missing namespace")
    for f in s.get("fields", []):
        name = f["name"]
        if not re.fullmatch(r"[a-z][a-z0-9_]*", name):
            errs.append(f"{name}: not snake_case")
        if "doc" not in f:
            errs.append(f"{name}: missing doc")
        # every non-first field should be optional with a default
        t = f["type"]
        is_union_with_null = isinstance(t, list) and "null" in t
        if "default" not in f and not is_union_with_null:
            errs.append(f"{name}: no default (evolution risk)")
    return errs

if __name__ == "__main__":
    bad = False
    for p in sys.argv[1:]:
        for e in lint(p):
            print(f"{p}: {e}"); bad = True
    sys.exit(1 if bad else 0)

// A field that fails two rules: camelCase name AND no default/doc
{"name": "orderTotal", "type": "long"}

Step-by-step explanation.

The linter loads each schema and checks structural house rules the registry never sees: a present namespace, snake_case field names, a doc string per field, and a default (or nullable union) on every field so future removals stay safe.
The offending field orderTotal fails three ways: orderTotal is camelCase (should be order_total), it has no doc, and it has no default.
The lint runs before the compatibility check in the same PR job, so style problems are reported alongside compatibility problems and the author fixes both at once.
The "every field has a default" rule is the highest-value lint: it pre-empts the entire class of "we added a required field and broke a consumer" incidents by making defaults mandatory.
Lints are house policy, so they live next to the schemas in git and evolve with the team — unlike the compatibility rules, which the registry owns.

Output.

Field	Violations	Fix
orderTotal (long)	camelCase, no doc, no default	order_total, add doc, add default
order_total (long, doc, default 0)	none	—

Rule of thumb. Lint what the registry cannot: naming, docs, and mandatory defaults. The mandatory-default lint alone removes most breaking-change risk before compatibility checking even runs.

Worked example — ownership and RBAC

Detailed explanation. Governance fails without a single accountable owner and enforced access. Record the owner in schema metadata and scope write access so only the owning team's CI can register. Walk through Confluent RBAC plus an ownership label.

Ownership. A owner label in the schema doc / metadata and in a CODEOWNERS file over schemas/.
Access. RBAC role granting Subject:Write on orders-* only to the payments-platform group.
Effect. Only the owner's pipeline can register; PRs need the owner's review.

Question. Assign ownership and lock write access to the owning team.

Input.

Control	Mechanism
repo ownership	CODEOWNERS over schemas/orders-*.avsc
registry write	RBAC role: Subject:Write on orders-*
consumer access	Subject:Read for consumer groups
audit	schema metadata owner label

Code.

# CODEOWNERS — PRs touching these schemas require the owning team's review
schemas/orders-*.avsc      @payments-platform
schemas/payments-*.avsc    @payments-platform

# Confluent RBAC — only payments-platform CI may WRITE orders-* subjects
confluent iam rbac role-binding create \
  --principal Group:payments-platform \
  --role DeveloperWrite \
  --resource "Subject:orders-" --prefix \
  --kafka-cluster-id "$CLUSTER" --schema-registry-cluster-id "$SR"

# Consumers get read-only on the subject
confluent iam rbac role-binding create \
  --principal Group:analytics-consumers \
  --role DeveloperRead \
  --resource "Subject:orders-" --prefix \
  --kafka-cluster-id "$CLUSTER" --schema-registry-cluster-id "$SR"

Step-by-step explanation.

CODEOWNERS makes every PR touching an orders-* schema require review from @payments-platform, so the human approval half of ownership is enforced by the repo, not by convention.
The Confluent RBAC DeveloperWrite binding, scoped by the orders- subject prefix, means only the payments-platform principal (its CI identity) may register or change those subjects — even a well-meaning consumer team cannot mutate the contract.
Consumers get DeveloperRead on the same prefix: they can fetch schemas to deserialize but not change them. This encodes "producer owns the schema, consumers subscribe" directly into access control.
The owner is also recorded in schema metadata (a label or a doc line) so an on-call engineer inspecting a subject can instantly find the accountable team.
Together, CODEOWNERS (review), RBAC write-scoping (registration), and read-only for consumers (subscription) make ownership real rather than aspirational.

Output.

Principal	Access to orders-*	Can register?
payments-platform (owner)	write	yes (via CI)
analytics-consumers	read	no
other teams	none	no
on-call	read + owner label	no

Rule of thumb. Encode ownership in three places: CODEOWNERS (who reviews), RBAC/IAM write-scoping (who registers), and schema metadata (who to page). A subject without a single accountable owner will rot exactly like the wiki it replaced.

Senior interview question on schema governance

A senior interviewer might ask: "Eight teams share your Kafka platform and last quarter two outages were caused by producers shipping breaking schema changes. Design an end-to-end governance workflow — repo, CI, registration, ownership, access control — that makes a breaking change impossible to ship, without slowing teams to a crawl."

Solution Using schema-as-code with a CI compatibility gate, RBAC, and clear ownership

Workflow (schema-as-code)
=========================
1. Schemas live in git under schemas/{subject}.avsc, owned via CODEOWNERS.
2. PR opens -> CI runs: schema-lint (naming/doc/default) THEN
   /compatibility check (no register). Either failure blocks the PR.
3. Owning-team review required (CODEOWNERS). Merge to main.
4. Merge job registers the reviewed version (auto.register.schemas=false in prod).
5. RBAC: only the owning team's CI has Subject:Write; consumers Subject:Read.
6. Deprecation: mark field deprecated in doc, window, then remove (default present).

# CI, both gates, before any registration
jobs:
  gate:
    steps:
      - run: python schema_lint.py schemas/*.avsc          # house rules
      - run: ./check_compat.sh schemas/*.avsc              # /compatibility, no register
  # register job runs only on push to main, after review

# Prod producers can never sneak a schema in
# application.properties
auto.register.schemas=false
use.latest.version=true

Step-by-step trace.

Layer	Control	Prevents
Repo	CODEOWNERS review	un-owned changes
CI lint	naming/doc/default rules	missing-default breaks
CI compat	/compatibility, no register	incompatible schema merging
Merge	register reviewed version	speculative schemas in registry
Runtime	auto.register.schemas=false	producer runtime sneak-in
RBAC	Subject:Write to owner only	wrong-team registration

After the workflow lands, a breaking change cannot ship: a missing-default or narrowing change fails the lint or the compatibility gate in the author's own PR; only reviewed, compatible schemas merge and register; production producers cannot auto-register; and only the owning team's CI identity holds write access. Teams still move fast because additive, default-carrying changes sail through green in seconds — the gate only bites on genuinely breaking edits.

Output:

Metric	Before	After
Breaking-change outages / quarter	2	0
Where breaks are caught	production	author's PR
Registration path	ad-hoc / runtime	CI on merge only
Schema source of truth	registry only	git (registry is a target)
Team velocity on safe changes	normal	unchanged (green in seconds)

Why this works — concept by concept:

Schema-as-code in git — schemas get review, history, and rollback, and the registry becomes a deployment target rather than the editing surface, so every change is a reviewable diff.
CI lint plus compatibility gate — the lint enforces mandatory defaults and naming (pre-empting most breaks) and the /compatibility check proves the change is safe without registering, so a breaking edit fails the author's own build.
Register-on-merge with auto-register off — only reviewed, merged schemas enter the registry, and production producers physically cannot register at runtime, closing the sneak-in path.
RBAC write-scoping and CODEOWNERS — ownership is real: one team reviews and one CI identity registers, so no other team can mutate the contract.
Cost — a CI job (a few seconds on safe changes), a git repo of schemas, and one-time RBAC setup. The eliminated cost is two production outages a quarter and the cross-team firefighting they trigger. O(subjects) config, O(1) per PR — cheap insurance against the most common Kafka-platform outage.

SQL
Topic — data-validation
Data-validation and governance-gate problems

Practice →

SQL
Topic — design
Design problems on data contracts and ownership

Practice →

Cheat sheet — Kafka Schema Registry recipes

Which registry when. Confluent is the 2026 default when you are self-managed and want the richest ecosystem. Apicurio when you need Apache-2.0 licensing, self-hosting, or multi-format artifacts (OpenAPI/AsyncAPI) — and it drops in via the Confluent-compatible ccompat API. AWS Glue when you are AWS-native (MSK/Kinesis/Lambda) and want IAM-governed access with no extra service to run. Keep every schema Avro-first so the choice stays reversible.
Compatibility mode decision map. BACKWARD (default) = new schema reads old data → upgrade consumers, then producers; legal: delete field, add optional field (with default). FORWARD = old schema reads new data → upgrade producers first; legal: add field, delete optional field. FULL = both → either side first; legal: only add/remove optional fields. *_TRANSITIVE = check against all prior versions, not just the latest — use for long-lived subjects with lagging consumers. NONE = no gate; greenfield only.
Avro evolution legal/illegal. Legal: add field with default, remove field that had a default, widen type (int→long, float→double), add enum symbol with a default-for-unknown, rename via alias. Illegal under BACKWARD: add required field (no default), remove required field, narrow type (long→int), change string↔int. Make "every optional field has a default" a CI lint failure.
Confluent wire format + REST. Wire = [0x00][4-byte big-endian global schema ID][payload]. Storage = compacted _schemas topic. REST: POST /subjects/{s}/versions (register), POST /compatibility/subjects/{s}/versions/latest (check without register), PUT /config/{s} (per-subject mode), GET /schemas/ids/{id} (consumer lookup). Version is per-subject; the ID on the wire is registry-global.
Subject-naming strategy picker. TopicNameStrategy (default) = {topic}-value, one record type per topic. RecordNameStrategy = subject is the record FQN → many event types on one topic (keep per-key ordering). TopicRecordNameStrategy = {topic}-{record-fqn} → many types per topic and the same type evolving differently across topics.
Apicurio ccompat swap. Change only schema.registry.url to http://apicurio:8080/apis/ccompat/v7; keep KafkaAvroSerializer. Wire format is identical (magic + 4-byte ID), so no code and no re-serialisation. Set compatibility via the ccompat /config route; Apicurio stores it as a COMPATIBILITY rule on the artifact.
Glue serializer config. Use GlueSchemaRegistryKafkaSerializer with registry.name, dataFormat=AVRO, compatibility=BACKWARD, region, and schemaAutoRegistrationEnabled=false in prod. Access is IAM: grant glue:GetSchemaVersion (consumers) and glue:RegisterSchemaVersion (producer CI) on the registry/schema ARNs. Glue's wire header is a UUID version id — moving to Glue is a serializer swap and a dual-publish, not a URL change.
CI compatibility-gate template. On every schema PR: run schema-lint (naming, docs, mandatory defaults) then POST /compatibility/.../versions/latest and fail the build if is_compatible != true. Register only on merge to main. In prod set auto.register.schemas=false and use.latest.version=true so producers can never register at runtime. A break becomes a red PR, not an outage.
Governance / ownership checklist. One accountable owner per subject (the producing team). Ownership encoded in CODEOWNERS (review), RBAC/IAM write-scoping (registration), and schema metadata (who to page). Consumers get read-only. Schemas live in git as code; the registry is a deployment target. Deprecate a field in its doc with a window before removing it (default already present).
Debugging deserialization. Read the first five bytes: a wrong magic byte means the record was not registry-framed; an unknown schema ID means the consumer cannot find the writer schema (registry down, wrong registry, or the ID was written against a different registry). For Glue, the header is a UUID, not a 4-byte int — a Confluent consumer against Glue bytes will fail on the magic byte.
Format notes. Avro = the reference, best resolution rules, logical types (timestamps, decimals). Protobuf = strong tooling, forward/backward via field numbers, first-class in Confluent/Apicurio. JSON Schema = human-readable, weakest binary efficiency. The registry enforces compatibility per format; the rules differ subtly, so do not assume Avro's add-field rule maps identically to Protobuf's field-number rule.

Frequently asked questions

What is a Kafka Schema Registry and why do I need one?

A Kafka Schema Registry is a versioned, compatibility-checked store of the schemas producers use to serialize records; the producer writes only a small schema ID onto the wire, and the consumer fetches the exact writer schema by ID to deserialize against its own reader schema. You need one whenever a topic has more than one consumer or more than one deploy cadence, because it turns the implicit "everyone agrees on the byte layout" contract — which drifts and breaks silently — into an explicit, enforced one. The registry also refuses to register a schema that would break the configured compatibility mode, so a producer physically cannot ship a change that a consumer cannot read. Without it you either embed the whole schema in every message (huge overhead) or rely on tribal knowledge that fails at 3 AM.

What are the compatibility modes and which should I pick?

The main modes are BACKWARD (the Confluent default — a new schema can read data written by the previous schema, so you upgrade consumers first-safely then producers), FORWARD (old consumers can read new data, so producers can lead), FULL (both hold; either side may deploy first), and NONE (no checking). Each also has a TRANSITIVE variant that checks a new version against all prior versions rather than only the latest. Pick BACKWARD (or BACKWARD_TRANSITIVE for long-lived subjects with lagging consumers) as the default, because the common risk is a producer racing ahead of slow consumers; move to FULL when both directions matter and you can restrict changes to optional-field adds and removes. Never run NONE on a subject that has real consumers.

Confluent vs Apicurio vs Glue — how do I choose?

Choose from constraints, not brand familiarity. Confluent Schema Registry is the reference implementation with the richest ecosystem — the default when you are self-managed and not AWS-locked. Apicurio is the open-source (Apache-2.0), self-hostable, multi-format alternative that also exposes a Confluent-compatible ccompat API, so existing Confluent serializers work against it with only a URL change — pick it for OSS mandates, multi-cloud, or OpenAPI/AsyncAPI artifacts. AWS Glue Schema Registry is AWS-native and IAM-governed, integrating directly with MSK, Kinesis, and Lambda — pick it when you are all-in on AWS and want schema access folded into IAM. Keep schemas Avro-first so a later switch is mostly configuration; note that moving to Glue changes the wire header and therefore requires a serializer swap, not just a URL change.

What is a subject-naming strategy?

A subject-naming strategy decides how the registry scopes compatibility — that is, what "unit" a schema evolves within. TopicNameStrategy (the default) names the subject {topic}-value (and {topic}-key), which means one record type per topic and compatibility scoped to that topic. RecordNameStrategy names the subject after the record's fully-qualified name, which lets several event types share one topic (useful when you must keep, say, OrderPlaced/OrderShipped/OrderCancelled in per-order order on one partition) with each type versioned independently. TopicRecordNameStrategy combines both, so the same record type can evolve differently on different topics. Use the default unless you genuinely need multiple event types on one topic, in which case consumers must dispatch on record type.

Avro vs Protobuf vs JSON Schema — does the registry care?

Yes — modern Confluent and Apicurio support all three, but the compatibility rules differ per format, so you cannot assume Avro's behaviour transfers. Avro is the historical reference: its writer-schema/reader-schema resolution maps cleanly onto the registry model, it supports logical types (timestamps, decimals), and its evolution rules (add field with default, widen types) are the best-documented. Protobuf evolves via field numbers rather than defaults and has its own forward/backward rules; it has strong cross-language tooling. JSON Schema is the most human-readable but the least space-efficient on the wire. Whatever you choose, the registry still enforces a compatibility mode — just verify the legal-change list for that format rather than reusing Avro's.

How do I stop a producer from shipping a breaking schema change?

Put a compatibility check in CI and register only on merge. On every pull request that touches a schema, call the registry's /compatibility endpoint (which answers "would this register?" without registering) and fail the build if the change is incompatible — so the break surfaces in the author's own PR, not in production. Register the reviewed schema only after the PR merges, and set auto.register.schemas=false on production producers so a running app can never sneak a new schema in at runtime. Add a lint that rejects fields without defaults and enforce single-team ownership through CODEOWNERS and RBAC/IAM write-scoping. Together these make a breaking change a red build instead of a 3 AM incident, while additive default-carrying changes still sail through green in seconds.

Practice on PipeCode

Drill the streaming practice library → for the Kafka schema-registry, compatibility-mode, and serialization problems senior interviewers love.
Rehearse on the event-processing practice library → for the multi-event-topic, subject-naming, and consumer-dispatch patterns.
Sharpen the schema axis with the JSON practice library → and the data-validation practice library → for schema-evolution and contract-check scenarios.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the registry-and-compatibility decision matrix against real graded inputs.

Lock in schema-registry muscle memory

Docs explain the endpoints. PipeCode drills explain the decision — when BACKWARD protects consumers, when a non-transitive check lets a v1 consumer drift, when RecordNameStrategy earns its place, when Apicurio ccompat is a URL change and when Glue is a serializer swap, and how a CI compatibility gate turns a breaking change into a red PR. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice streaming problems →
Practice data-validation problems →

1. Why the schema registry is the contract layer of a Kafka platform

The registry stores writer schemas by subject and puts an ID on the wire — that indirection is what lets producers and consumers evolve independently

Worked example — the four-axis comparison table

Worked example — what interviewers actually probe

Worked example — the "pick the registry" decision tree

Senior interview question on schema-registry design

Solution Using a Confluent registry with BACKWARD compatibility and a CI gate

2. Compatibility modes and schema evolution

compatibility modes are the rulebook the registry enforces on every new version — and they encode which side of the pipeline may upgrade first

Worked example — adding an optional field under BACKWARD

Worked example — the transitive-drift trap

Worked example — an illegal change the registry rejects

Senior interview question on compatibility modes

Solution Using BACKWARD_TRANSITIVE with default-carrying optional fields

3. Confluent Schema Registry deep dive

Confluent Schema Registry is the reference implementation — schemas live in a compacted _schemas topic, the wire carries a magic byte plus a 4-byte schema ID, and subject naming decides scope

Worked example — register and evolve a subject via REST

Worked example — serializer config and the wire bytes

Worked example — RecordNameStrategy for multi-event topics

Senior interview question on Confluent internals

Solution Using the KafkaAvroSerializer end-to-end with an ID cache

4. Apicurio and AWS Glue Schema Registry

Apicurio is the open-source, Confluent-compatible registry and AWS Glue Schema Registry is the AWS-native, IAM-governed one — both speak the same producer contract if you keep it Avro-first

Worked example — swapping Confluent for Apicurio via ccompat

Worked example — Glue serializer config with IAM governance

Worked example — the three-registry decision under real constraints

Senior interview question on registry choice and interop

Solution Using Apicurio ccompat for the OSS mandate and Glue for the AWS unit

5. Governance, ownership, and CI enforcement

governance is what turns a registry from storage into a control plane — CI compatibility gates, clear ownership, and access control stop a breaking change before it reaches a running consumer

Worked example — a CI compatibility gate in the pipeline

Worked example — schema-lint rules beyond compatibility

Worked example — ownership and RBAC

Senior interview question on schema governance

Solution Using schema-as-code with a CI compatibility gate, RBAC, and clear ownership

Cheat sheet — Kafka Schema Registry recipes

Frequently asked questions

What is a Kafka Schema Registry and why do I need one?

What are the compatibility modes and which should I pick?

Confluent vs Apicurio vs Glue — how do I choose?

What is a subject-naming strategy?

Avro vs Protobuf vs JSON Schema — does the registry care?

How do I stop a producer from shipping a breaking schema change?

Practice on PipeCode

Lock in schema-registry muscle memory

`compatibility modes` are the rulebook the registry enforces on every new version — and they encode which side of the pipeline may upgrade first

`Confluent Schema Registry` is the reference implementation — schemas live in a compacted `_schemas` topic, the wire carries a magic byte plus a 4-byte schema ID, and subject naming decides scope

`Apicurio` is the open-source, Confluent-compatible registry and `AWS Glue Schema Registry` is the AWS-native, IAM-governed one — both speak the same producer contract if you keep it Avro-first

`governance` is what turns a registry from storage into a control plane — CI compatibility gates, clear ownership, and access control stop a breaking change before it reaches a running consumer