Avro vs Protobuf vs JSON Schema: Serialization & Schema Evolution for Streaming

avro vs protobuf is the pick-one decision that quietly decides whether your Kafka topics stay readable for years or turn into an undeployable minefield the first time a producer adds a field — and it is the single design choice most data engineers make by copying whatever the last team used, without ever understanding the three axes that actually separate the formats. Every event your pipeline emits — an order placed, a click logged, a feature vector recomputed — has to be turned into bytes by a producer, shipped across a broker, and turned back into a typed record by a dozen consumers that were built at different times, deployed on different days, and each carry a different idea of what the record's fields are. The engineering problem is not "should we serialize" — every streaming stack must — but which wire format you pick and how it lets the record's shape change underneath a running fleet without breaking a single consumer.

This guide is the walkthrough you wished existed the first time an interviewer asked "compare avro vs protobuf for a Kafka pipeline," or "what actually happens on the wire when a producer serializes an Avro record against Confluent Schema Registry?", or "your producer adds a required field and every consumer starts crashing — walk me through why, and which compatibility mode would have caught it in CI." It works through the three formats — Avro (schema-on-read resolution, the reader-schema trick, registry framing), Protobuf (field-number tags, varint wire format, proto3 presence), and JSON Schema (human-readable validation layered over plain JSON) — the three axes that separate them (where the schema lives, how many bytes hit the wire, and which schema-evolution rules govern safe change), and the head-to-head compatibility matrix — backward, forward, full, transitive — that turns "we hope this deploy is safe" into a CI check. Each section pairs a teaching block with a Solution-Tail interview answer: runnable code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse JSON parsing on the JSON practice library →, and sharpen the pipeline axis with the ETL practice library →.

---

On this page

• Why serialization format is a load-bearing streaming choice
• Avro — writer/reader schema resolution and the registry
• Protobuf — field numbers, varint wire format, proto3
• JSON Schema — human-readable validation vs serialization
• Head-to-head — compatibility, registry, decision matrix
• Cheat sheet — serialization & schema evolution recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why serialization format is a load-bearing streaming choice

Three axes — schema location, wire size, evolution rules — decide the format, and the choice binds every consumer for years

The one-sentence invariant: a streaming serialization format is a pick-one trade-off across three axes — where the schema physically lives (embedded in each message, referenced by an ID in a registry, or absent entirely), how many bytes each record costs on the wire, and which schema-evolution rules govern whether adding or removing a field breaks a running consumer — and the format you pick in month one becomes the format every downstream consumer, sink connector, and replay job hard-codes assumptions about for years. avro vs protobuf vs json schema is not a taste debate; it is a decision about which of the three axes your pipeline is most sensitive to, and getting it wrong ships either a bloated wire bill (JSON everywhere), an undecodeable topic (Avro with no registry), or a silent field-collision bug (Protobuf field-number reuse) that only surfaces during an incident.

The three axes, defined precisely.

• Schema location. Avro data carries no field names and no types on the wire — it is a positional stream of values, so you cannot decode a byte without the writer's schema. That schema lives in a schema registry, referenced by a 4-byte ID prepended to each message. Protobuf carries field numbers (not names) as tags on the wire, so it is partially self-describing — you can skip unknown fields without the .proto, but you cannot recover field names or types without it. JSON Schema sits over plain JSON, which is fully self-describing text — the field names and value shapes are in the payload, and the schema is used only for validation, never for decoding.
• Wire size. For fully-populated records Avro is usually the smallest — zero per-field tag overhead, positional values, varint integers. Protobuf is close behind and wins on sparse records because unset fields cost zero bytes. JSON is the largest by a wide margin — every field name is repeated as UTF-8 text in every single message, and numbers are stored as decimal strings.
• Evolution rules. Avro resolves a writer schema against a reader schema using field names and defaults. Protobuf resolves by field number and preserves unknown fields. JSON Schema evolves by loosening or tightening required and additionalProperties. Each format has a different definition of "a safe change," and that definition is the axis interviewers probe hardest.

Why the choice is irreversible in practice.

• Every consumer hard-codes the deserializer. A topic serialized with the Confluent Avro serializer is decoded by consumers that instantiate an AvroDeserializer wired to the registry. Switching the topic to Protobuf means every consumer redeploys with a new deserializer and a coordinated cutover — you cannot mix formats in one topic partition without a discriminator.
• Replay and reprocessing assume the format. Tiered-storage replays, Kafka-to-warehouse sink connectors, and disaster-recovery re-reads all decode historical bytes with the historical schema. A format migration orphans years of retained data unless you dual-write or re-encode.
• The registry is the contract. Once subject orders-value has 40 registered Avro versions, the compatibility history is the API contract. Consumers trust that version N+1 is BACKWARD compatible with version N; break that trust once and you break every consumer that upgraded on that promise.

What interviewers listen for.

• Do you name all three axes — schema location, wire size, evolution — without prompting? — senior signal.
• Do you say "Avro data is undecodeable without the writer schema" the moment Avro comes up? — required answer.
• Do you push back on "just use JSON, it's simpler" with the wire-cost and no-enforcement arguments? — senior signal.
• Do you describe the schema registry as "the contract enforcement point," not "a place to store schemas"? — senior signal.
• Do you distinguish backward compatibility (new reader reads old data) from forward compatibility (old reader reads new data) in one clean sentence? — required answer.

Worked example — the same record in three wire formats

Detailed explanation. The single most clarifying exercise for a serialization interview is to encode one identical record in all three formats and count the bytes. Every abstract claim about wire size becomes concrete the moment you see {"id":42,"name":"Ada","active":true} cost 36 bytes as JSON, 6 bytes as Avro, and 9 bytes as Protobuf. Walk through the record and the mental model for each format's overhead.

• The record. id=42 (a small integer), name="Ada" (a 3-character string), active=true (a boolean).
• JSON overhead. Every field name (id, name, active) is repeated as text in every message, plus braces, quotes, colons, commas.
• Avro overhead. None per field — values are written positionally in schema-declared order; the integer is a 1-byte zig-zag varint.
• Protobuf overhead. One tag byte per present field (field number + wire type), then the value.

Question. Encode {id: 42, name: "Ada", active: true} in JSON, Avro, and Protobuf and report the byte count and the reason for each.

Input.

Field	Value	JSON bytes	Avro bytes	Protobuf bytes
id	42	"id":42 = 7	zig-zag varint = 1	tag 08 + varint 2a = 2
name	"Ada"	"name":"Ada" = 13	len 06 + Ada = 4	tag 12 + len 03 + Ada = 5
active	true	"active":true = 14	01 = 1	tag 18 + 01 = 2
structural	braces/commas	{,,} = 3 (approx)	0	0
total		36	6	9

Code.

# byte_size_comparison.py — encode one record in three formats, count bytes
import io
import json
import fastavro          # pip install fastavro
import user_pb2          # generated: protoc --python_out=. user.proto

record = {"id": 42, "name": "Ada", "active": True}

# --- JSON ---
json_bytes = json.dumps(record, separators=(",", ":")).encode("utf-8")

# --- Avro (schemaless, no registry framing) ---
avro_schema = fastavro.parse_schema({
    "type": "record", "name": "User", "namespace": "com.pipecode",
    "fields": [
        {"name": "id",     "type": "long"},
        {"name": "name",   "type": "string"},
        {"name": "active", "type": "boolean"},
    ],
})
buf = io.BytesIO()
fastavro.schemaless_writer(buf, avro_schema, record)
avro_bytes = buf.getvalue()

# --- Protobuf ---
proto_bytes = user_pb2.User(id=42, name="Ada", active=True).SerializeToString()

for name, b in [("JSON", json_bytes), ("Avro", avro_bytes), ("Protobuf", proto_bytes)]:
    print(f"{name:9} {len(b):3} bytes  {b.hex()}")

Step-by-step explanation.

1. JSON serializes to text: even with the compact separators=(",", ":") (no spaces), the three field names alone cost 15 bytes that repeat in every message. On a topic doing a million events/second, those repeated names are the dominant cost line.
2. Avro's schemaless_writer writes only values, in the order the schema declares them. 42 becomes a single zig-zag varint byte (0x54), the string is a 1-byte length prefix plus 3 UTF-8 bytes, the boolean is one byte. Six bytes total — but the six bytes are meaningless without the schema that says "byte 0 is a long named id."
3. Protobuf writes a tag byte before each field: 0x08 is field 1 as a varint, 0x12 is field 2 as a length-delimited value, 0x18 is field 3 as a varint. The tag encodes (field_number << 3) | wire_type, so field names never touch the wire — only numbers do.
4. The key asymmetry: Avro's 6 bytes assume the reader has the schema out-of-band; Protobuf's 9 bytes are partially self-describing (a decoder can skip field 4 it has never seen). That extra 3 bytes buys forward-skip resilience without a registry.
5. Scale the comparison mentally: at a billion events/day, JSON's 36 bytes vs Avro's 6 is a 6× storage-and-network multiplier — the axis that pushes high-volume telemetry pipelines off JSON regardless of its readability.

Output.

JSON      36 bytes  7b226964223a34322c226e616d65223a22416461222c226163746976653a747275657d
Avro       6 bytes  54064164610 1        (0x54 0x06 41 64 61 01)
Protobuf   9 bytes  082a1203416461 1801  (08 2a 12 03 41 64 61 18 01)

Rule of thumb. Never argue serialization format from readability alone. Encode a representative record, count the bytes, multiply by your event rate, and put the wire-cost number next to the human-readability benefit. For high-volume topics the 6× JSON tax usually decides it; for low-volume debuggable APIs the readability usually wins.

Worked example — the decode-dependency table

Detailed explanation. The axis interviewers probe first is "what do you need in hand to turn these bytes back into a record?" Each format answers differently, and the answer determines your operational blast radius when the registry is down or a schema is missing. Build the decode-dependency table.

• Avro. You need the writer schema (the exact schema used to encode). With Confluent framing, you fetch it from the registry by the 4-byte ID in the message. No schema, no decode — full stop.
• Protobuf. You can skip unknown fields and read known field numbers without the .proto, but you cannot recover types or names. For a typed record you need the compiled descriptor.
• JSON Schema. You need nothing to decode — JSON is self-describing. You need the schema only to validate that the decoded object obeys the contract.

Question. For each format, state what is required to (a) decode the bytes into a usable structure and (b) validate the structure against the contract.

Input.

Format	Bytes are self-describing?	Needed to decode	Needed to validate
Avro	no (positional)	writer schema (via registry ID)	reader schema
Protobuf	partially (field numbers only)	compiled .proto descriptor	the .proto field rules
JSON Schema	yes (names + shapes in text)	nothing (any JSON parser)	the JSON Schema document

Code.

# decode_dependency.py — what each format needs to become a Python object
import io
import json
import struct
import fastavro
import user_pb2

# Confluent-framed Avro message: magic(1) + schema_id(4) + avro_payload
def decode_avro_confluent(frame: bytes, registry: dict) -> dict:
    magic = frame[0]
    if magic != 0:
        raise ValueError("not a Confluent-framed message")
    schema_id = struct.unpack(">I", frame[1:5])[0]   # 4-byte big-endian ID
    writer_schema = registry[schema_id]              # MUST look this up
    return fastavro.schemaless_reader(io.BytesIO(frame[5:]), writer_schema)

# Protobuf: needs the compiled descriptor, but can skip unknowns
def decode_proto(raw: bytes) -> user_pb2.User:
    msg = user_pb2.User()
    msg.ParseFromString(raw)      # unknown fields are preserved, not lost
    return msg

# JSON: needs nothing to decode; schema only validates
def decode_json(raw: bytes) -> dict:
    return json.loads(raw)        # self-describing; no schema required

Step-by-step explanation.

1. decode_avro_confluent shows the hard dependency: the first byte must be the magic 0x00, the next four are the schema ID, and everything downstream fails if the registry lookup misses. This is why an Avro topic with a purged registry is unreadable — the bytes are values with no labels.
2. decode_proto needs the generated user_pb2 module (the compiled descriptor) to produce a typed User. But ParseFromString will happily consume a message containing field 4 it has never seen — the unknown bytes are retained in the message's unknown-field set, which is what makes Protobuf forward-skip safe.
3. decode_json needs nothing but a JSON parser. The trade-off is that "nothing enforces the contract" — a consumer that expects id to be an integer will happily receive "id": "oops" and only discover the problem when arithmetic on it throws.
4. The validation column is orthogonal to the decode column. Avro's reader schema both decodes and validates in one pass (resolution rejects incompatible types). JSON Schema splits the two: json.loads decodes, and a separate jsonschema.validate enforces the contract.
5. Operationally, the decode-dependency table is your incident runbook. Registry down → Avro topics stall, Protobuf consumers keep running on cached descriptors, JSON consumers are unaffected. Knowing this before the incident is the senior signal.

Output.

Failure	Avro	Protobuf	JSON Schema
Registry unavailable	topic undecodeable	keeps running (local descriptor)	unaffected
Unknown new field	resolved via schema	preserved, skipped	present in dict
Wrong value type	rejected at resolution	rejected at parse	silently accepted (unless validated)

Rule of thumb. Map every format to its decode dependency before you pick one. If your operational reality includes "the registry can be down and consumers must keep working," that constraint alone pushes you toward Protobuf or JSON over registry-coupled Avro.

Worked example — backward vs forward compatibility in one picture

Detailed explanation. The two words interviewers most want you to define precisely are backward compatibility and forward compatibility. They are directional, they are about who upgrades first, and mixing them up is the fastest way to fail a schema-evolution question. Nail the definitions with a concrete deploy order.

• Backward compatible. A new reader can read data written by an old writer. You upgrade consumers first. This is the safe default because you rarely control when every producer upgrades.
• Forward compatible. An old reader can read data written by a new writer. You upgrade producers first. Useful when producers must ship a new field before consumers are ready.
• Full compatible. Both directions hold — you can deploy producers and consumers in any order. The most restrictive, the safest, the slowest to earn.

Question. Given "add a nullable email field with a default," classify the change under each compatibility direction and state the safe deploy order.

Input.

Direction	Who upgrades first	"Add field with default" verdict
Backward	consumers	safe (new reader supplies default for old data)
Forward	producers	safe (old reader ignores the new field)
Full	any order	safe (holds both directions)

Code.

# compatibility_direction.py — illustrate the two directions with Avro resolution
import io
import fastavro

writer_v1 = fastavro.parse_schema({
    "type": "record", "name": "User", "namespace": "com.pipecode",
    "fields": [{"name": "id", "type": "long"}, {"name": "name", "type": "string"}],
})

reader_v2 = fastavro.parse_schema({
    "type": "record", "name": "User", "namespace": "com.pipecode",
    "fields": [
        {"name": "id",    "type": "long"},
        {"name": "name",  "type": "string"},
        {"name": "email", "type": "string", "default": ""},   # NEW, has default
    ],
})

# Old writer emits v1 bytes
buf = io.BytesIO()
fastavro.schemaless_writer(buf, writer_v1, {"id": 7, "name": "Grace"})
old_bytes = buf.getvalue()

# BACKWARD: new reader (v2) reads old (v1) bytes — supplies default for email
buf.seek(0)
record = fastavro.schemaless_reader(buf, writer_v1, reader_v2)
print(record)   # {'id': 7, 'name': 'Grace', 'email': ''}

Step-by-step explanation.

1. writer_v1 has two fields; reader_v2 adds email with a default of "". The default is the load-bearing detail: it tells the reader what to fill in when the field is absent from the old bytes.
2. schemaless_reader(buf, writer_v1, reader_v2) performs Avro schema resolution — it matches id and name by name, finds no email in the writer schema, and supplies the reader's default. This is backward compatibility working: the new reader read old data.
3. Forward compatibility is the mirror image: an old reader (v1) reading new (v2) bytes simply ignores the extra email field it does not know about. Because the added field has a default, the change is full compatible — both directions hold.
4. Remove the default and the story changes: adding a field without a default is not backward compatible, because a new reader hitting old data has no value to fill in and resolution fails. This is the single most common schema-evolution mistake.
5. The deploy order falls directly out of the direction. Backward-compatible change → upgrade consumers first. Forward-compatible change → upgrade producers first. Full → any order. Say the deploy order unprompted; it is the difference between a textbook answer and a shipped-it answer.

Output.

Change	Backward?	Forward?	Full?	Safe deploy order
Add field WITH default	yes	yes	yes	any order
Add field WITHOUT default	no	yes	no	producers first
Remove field WITH default	yes	no	no	consumers first
Remove field WITHOUT default	no	no	no	unsafe — needs a migration

Rule of thumb. Say the two definitions as "new reads old = backward, old reads new = forward" and always append the deploy order. A default value is what turns a risky add into a full-compatible one — attach a default to every new field by reflex.

Common beginner mistakes.

• Treating format choice as a style preference. The three axes — schema location, wire size, evolution — are engineering constraints, not opinions. Pick the axis your workload is most sensitive to first.
• Assuming JSON is free. JSON's readability hides a 3–8× wire tax versus Avro/Protobuf and zero contract enforcement unless you bolt on JSON Schema validation.
• Confusing backward and forward compatibility. They are directional and about deploy order. Mixing them up produces a plausible-sounding but wrong answer.
• Shipping an Avro topic with no registry story. Avro bytes are undecodeable without the writer schema; "we'll figure out the registry later" is a topic you cannot replay.
• Reusing a Protobuf field number. Field numbers are the permanent contract; reusing one silently misinterprets old bytes as a new field's value.

Streaming interview question on choosing a serialization format

A senior interviewer often opens with: "We're standing up a new Kafka pipeline: high-volume clickstream into a warehouse, plus a handful of typed domain events consumed by microservices. Producers and consumers are owned by different teams and deploy independently. Walk me through how you'd pick between avro vs protobuf vs JSON Schema, what enforcement you'd put in CI, and how you'd let schemas evolve without a coordinated deploy."

Solution Using a per-workload format choice with registry-enforced compatibility

# format_decision.py — encode the axis-driven decision as a function
from dataclasses import dataclass

@dataclass
class Workload:
    name: str
    events_per_sec: int
    consumers_independent: bool     # do producers/consumers deploy separately?
    debuggability_critical: bool    # do humans read raw messages routinely?
    typed_domain_events: bool       # microservice contracts, not just telemetry?

def pick_format(w: Workload) -> str:
    # Axis 1 (wire size) dominates at high volume
    if w.events_per_sec > 100_000 and not w.debuggability_critical:
        base = "avro"                      # smallest fully-populated records
    elif w.typed_domain_events:
        base = "protobuf"                  # strong typing + gRPC reuse
    elif w.debuggability_critical:
        base = "json-schema"               # readable, validated
    else:
        base = "avro"                      # registry default

    # Axis 3 (evolution) — independent deploys demand FULL compatibility
    compat = "FULL_TRANSITIVE" if w.consumers_independent else "BACKWARD"
    return f"{base} + {compat}"

clickstream = Workload("clickstream", 500_000, True,  False, False)
domain      = Workload("orders",       2_000,   True,  False, True)
audit       = Workload("audit-log",    50,      True,  True,  False)

for w in (clickstream, domain, audit):
    print(f"{w.name:12} -> {pick_format(w)}")

# ci-compatibility-check.yml — block a merge that breaks the contract
# Runs the Confluent Schema Registry maven/gradle compatibility test in CI
steps:
  - name: Register-check schema against subject
    run: |
      # Dry-run: does the new schema pass the subject's compatibility mode?
      curl -s -X POST \
        -H "Content-Type: application/vnd.schemaregistry.v1+json" \
        --data @schema-payload.json \
        "$SR_URL/compatibility/subjects/orders-value/versions/latest" \
        | jq -e '.is_compatible == true'   # exits non-zero (fails CI) if false

Step-by-step trace.

Step	Input	Decision
Clickstream volume	500k/s, not debug-critical	wire size dominates → Avro
Orders typing	typed domain events	strong typing → Protobuf
Audit debuggability	humans read it, 50/s	readability → JSON Schema
Deploy independence	all three teams deploy separately	FULL_TRANSITIVE compatibility
CI enforcement	dry-run POST to /compatibility	non-compatible schema fails the merge

The result is a per-workload format choice — not one format decreed for the whole company — with the schema registry as the enforcement point. The CI check turns "we hope this schema change is safe" into a gate: the compatibility endpoint returns is_compatible: false and the merge is blocked before a single broken byte reaches a broker.

Output:

Workload	Format	Compatibility	Why
clickstream	Avro	FULL_TRANSITIVE	6× smaller than JSON at 500k/s
orders (domain)	Protobuf	FULL_TRANSITIVE	typed contracts, gRPC reuse
audit-log	JSON Schema	FULL_TRANSITIVE	humans read raw messages
enforcement	registry /compatibility	fails CI on break	catch it before the broker

Why this works — concept by concept:

• Axis-driven choice — the decision is not "one format to rule them all" but "which axis does this workload live and die on." High-volume telemetry is wire-size-bound (Avro); typed microservice contracts are type-safety-bound (Protobuf); human-audited streams are readability-bound (JSON Schema).
• Schema registry as contract — registering every schema under a subject with a compatibility mode makes the history the API. Version N+1 is provably compatible with N because the registry refused to register it otherwise.
• FULL_TRANSITIVE for independent deploys — when producers and consumers deploy in any order, only full compatibility across all prior versions guarantees no ordering of deploys breaks. Backward-only would strand producers that ship ahead of consumers.
• CI compatibility gate — the POST /compatibility/subjects/.../versions/latest dry-run is the shift-left. It moves the failure from "3am pager, consumers crash-looping" to "red X on a pull request."
• Cost — one registry (cheap, HA), a per-message 5-byte framing overhead for Avro/Protobuf, and one CI job per schema change. The eliminated cost is the coordinated-deploy war-room and the undecodeable-topic incident. O(1) enforcement per schema change versus O(consumers) manual coordination.

Streaming
Topic — streaming
Streaming serialization and schema-registry problems

Practice →

JSON Topic — json JSON parsing and payload-shape problems

Practice →

---

2. Avro — writer/reader schema resolution and the registry

Avro is schema-on-read — the writer schema encodes, the reader schema resolves, and the registry is the glue that makes it work on Kafka

The mental model in one line: Avro serializes a record as a positional stream of values against a writer schema, and any consumer deserializes those bytes against its own reader schema — Avro's resolution algorithm reconciles the two by field name, filling reader-only fields from their defaults and dropping writer-only fields — which is exactly why Avro carries no field names on the wire (making it compact) and cannot be decoded without the writer schema (making the schema registry non-optional on Kafka). Every senior data engineer who runs Confluent Kafka has internalised the 5-byte framing that turns a raw Avro payload into a self-locating message, and every one has been burned once by adding a field without a default.

The three things that make Avro Avro.

• Positional binary encoding. Fields are written in schema-declared order with no names and no type tags. An int/long is a zig-zag varint; a string is a length-prefixed UTF-8 blob; a record is just its fields back-to-back. This is why Avro is compact and why the schema is mandatory to decode.
• Writer/reader schema resolution. The writer schema is what encoded the bytes; the reader schema is what the consumer wants. Resolution matches fields by name: a field in the reader but not the writer is filled from the reader's default; a field in the writer but not the reader is decoded-and-discarded. Type promotions (int→long, float→double) are allowed; incompatible ones fail.
• Defaults are the evolution primitive. Every field a future reader might add needs a default so that resolution can fill it when reading old data. "Add a field with a default" is the canonical backward-compatible change; "add a field without a default" is the canonical break.

Confluent Schema Registry framing — the 5 bytes that make Avro work on Kafka.

• Byte 0 — magic byte 0x00. A version marker for the framing itself.
• Bytes 1–4 — schema ID. A big-endian 32-bit integer that identifies the exact writer schema in the registry. The consumer reads these four bytes, fetches the writer schema (cached after the first lookup), and only then can it decode.
• Bytes 5+ — the Avro payload. The positional value stream, encoded with the writer schema referenced by the ID.
• Subjects and versions. Schemas register under a subject — by default <topic>-value (and <topic>-key). Each registration is a version; the registry enforces the subject's compatibility mode at registration time.

The reader-schema trick — project any past version into the shape you want.

• Read old data with a new shape. Point the deserializer at your current reader schema; resolution back-fills every field you added since, using defaults. You never rewrite historical bytes.
• Drop fields you no longer care about. A reader schema that omits a field the writer wrote decodes-and-discards it — a cheap projection at read time.
• Rename via aliases. Avro aliases let a reader field match a differently-named writer field, enabling safe renames without re-encoding.

Where Avro wins and loses.

• Wins. Smallest fully-populated records; first-class schema evolution with resolution + defaults; the native format of the Hadoop/Kafka/Confluent ecosystem; block-compressible Avro object container files for data-lake landing.
• Loses. Undecodeable without the writer schema (hard registry dependency); no partial self-description (a stray byte offset corrupts the rest of the record); more awkward in polyglot RPC than Protobuf.

Worked example — define a schema, serialize, and read it back

Detailed explanation. The canonical Avro round-trip: declare an .avsc, encode a record to bytes, decode it back. Doing it schemaless (no container-file header) mirrors what the Kafka serializer does inside the 5-byte frame. Walk through the round-trip and inspect the raw bytes.

• Schema. A User record with id: long, name: string, active: boolean (default true).
• Encode. schemaless_writer emits the positional value stream.
• Decode. schemaless_reader with the same schema reconstructs the dict.

Question. Serialize {"id": 42, "name": "Ada"} (omitting active) against a schema where active defaults to true, then decode and confirm the default was applied.

Input.

Field	Type	Default	Provided value
id	long	—	42
name	string	—	"Ada"
active	boolean	true	(omitted)

Code.

# avro_roundtrip.py
import io
import fastavro

schema = fastavro.parse_schema({
    "type": "record", "name": "User", "namespace": "com.pipecode",
    "fields": [
        {"name": "id",     "type": "long"},
        {"name": "name",   "type": "string"},
        {"name": "active", "type": "boolean", "default": True},
    ],
})

# Encode — fastavro applies the default for the omitted 'active'
buf = io.BytesIO()
fastavro.schemaless_writer(buf, schema, {"id": 42, "name": "Ada"})
raw = buf.getvalue()
print("bytes:", raw.hex(), "len:", len(raw))     # 5406416461 01 -> 6 bytes

# Decode — same schema acts as both writer and reader
buf.seek(0)
record = fastavro.schemaless_reader(buf, schema)
print("record:", record)                         # active back as True

Step-by-step explanation.

1. parse_schema compiles the .avsc dict into fastavro's internal form. The default: True on active is what lets the writer fill in a value for the omitted field at encode time and what a future reader would use at decode time.
2. schemaless_writer walks the fields in declared order: id → zig-zag varint (0x54), name → length 0x06 (zig-zag of 3) + "Ada", active → the default True as 0x01. No field names appear anywhere in the output.
3. The hex 54 06 41 64 61 01 is exactly 6 bytes and is meaningless in isolation — byte 0 is only "the long named id" because the schema says so. Hand these 6 bytes to a consumer without the schema and they cannot be decoded.
4. schemaless_reader(buf, schema) uses the same schema as both writer and reader, so resolution is trivial — read each field positionally. active comes back True because it was encoded (from its default) as 0x01.
5. On Kafka the Confluent serializer would prepend 00 00 00 00 <id> (magic + schema ID) to these 6 bytes; the deserializer strips those 5 bytes, looks up the writer schema by ID, and runs the exact same positional decode.

Output.

bytes: 540641646101 len: 6
record: {'id': 42, 'name': 'Ada', 'active': True}

Rule of thumb. Attach a default to every field you can, even required-looking ones. Defaults are Avro's evolution currency — a field without one is a field you cannot add backward-compatibly later, and you will want to add it later.

Worked example — schema evolution via writer/reader resolution

Detailed explanation. The reason Avro is beloved on Kafka is resolution: a consumer built today can read a message written last year by pointing its reader schema at the writer schema the message was encoded with. Walk through adding a field and reading old bytes through the new shape.

• Writer v1. id, name.
• Reader v2. id, name, email (default ""), and drops nothing.
• Goal. Decode v1 bytes through the v2 reader and confirm email is back-filled.

Question. Encode a record with writer v1, then decode it with reader v2 that added an email field, and show that resolution supplies the default.

Input.

Schema	Fields	Role
v1	id, name	writer (encoded the bytes)
v2	id, name, email (default "")	reader (wants three fields)

Code.

# avro_evolution.py
import io
import fastavro

writer_v1 = fastavro.parse_schema({
    "type": "record", "name": "User", "namespace": "com.pipecode",
    "fields": [{"name": "id", "type": "long"}, {"name": "name", "type": "string"}],
})

reader_v2 = fastavro.parse_schema({
    "type": "record", "name": "User", "namespace": "com.pipecode",
    "fields": [
        {"name": "id",    "type": "long"},
        {"name": "name",  "type": "string"},
        {"name": "email", "type": "string", "default": ""},
    ],
})

buf = io.BytesIO()
fastavro.schemaless_writer(buf, writer_v1, {"id": 7, "name": "Grace"})
buf.seek(0)

# Resolution: pass BOTH schemas. fastavro reconciles v1 bytes into the v2 shape.
record = fastavro.schemaless_reader(buf, writer_v1, reader_v2)
print(record)     # email supplied from its default

Step-by-step explanation.

1. writer_v1 encodes {"id": 7, "name": "Grace"} as two positional values — no room for email because the writer schema has no such field. The bytes are frozen at 2 fields forever.
2. reader_v2 adds email with a default. This default is the entire mechanism: it is the value resolution uses when the writer bytes have nothing for that field.
3. schemaless_reader(buf, writer_v1, reader_v2) runs Avro schema resolution: match id↔id, name↔name by name; email exists in the reader but not the writer, so fill it from the reader's default "".
4. Had email lacked a default, resolution would raise — a new reader hitting old data with no value to supply is the definition of a not-backward-compatible change. The registry's BACKWARD mode would have rejected v2 at registration time and blocked the deploy.
5. This is why "upgrade consumers first" is the safe order for backward-compatible changes: the new (v2) readers can already read the old (v1) producers' bytes, so consumers can roll out before producers do.

Output.

{'id': 7, 'name': 'Grace', 'email': ''}

Rule of thumb. Test evolution the way Avro decodes it — pass the old writer schema and the new reader schema to the deserializer and assert the result. If it resolves, the change is backward compatible; if it raises, you just found the break before your consumers did.

Worked example — parse the Confluent 5-byte wire frame by hand

Detailed explanation. Interviewers love asking "what are the first five bytes of an Avro message on Confluent Kafka?" because the answer proves you understand that Avro needs out-of-band schema resolution. Decode the frame manually.

• Byte 0. Magic byte, always 0x00.
• Bytes 1–4. Schema ID, big-endian uint32.
• Bytes 5+. The positional Avro payload.

Question. Given a Confluent-framed message, extract the schema ID and hand the payload to the correct writer schema fetched from the registry.

Input.

Byte range	Meaning	Example
[0]	magic byte	0x00
[1:5]	schema ID (big-endian)	00 00 00 2a → 42
[5:]	Avro payload	positional values

Code.

# confluent_frame.py — decode the wire framing without the Kafka client
import io
import struct
import fastavro

def deserialize_confluent_avro(frame: bytes, fetch_schema) -> dict:
    if frame[0] != 0:
        raise ValueError(f"unexpected magic byte {frame[0]:#x}; not Confluent Avro")
    schema_id = struct.unpack(">I", frame[1:5])[0]     # 4-byte big-endian ID
    writer_schema = fetch_schema(schema_id)            # registry lookup (cached)
    return fastavro.schemaless_reader(io.BytesIO(frame[5:]), writer_schema)

# --- simulate the registry + a produced frame ---
schema_42 = fastavro.parse_schema({
    "type": "record", "name": "User", "namespace": "com.pipecode",
    "fields": [{"name": "id", "type": "long"}, {"name": "name", "type": "string"}],
})
REGISTRY = {42: schema_42}

payload = io.BytesIO()
fastavro.schemaless_writer(payload, schema_42, {"id": 7, "name": "Grace"})
frame = b"\x00" + struct.pack(">I", 42) + payload.getvalue()

print("frame:", frame.hex())
print("decoded:", deserialize_confluent_avro(frame, REGISTRY.__getitem__))

Step-by-step explanation.

1. The producer's Confluent serializer registered schema_42 under the subject, got back ID 42, and prepended 00 00 00 00 2a — magic 0x00 then the big-endian uint32 42 — to the 5-byte Avro payload.
2. deserialize_confluent_avro first asserts frame[0] == 0. A non-zero magic byte means the message is not Confluent-framed (someone produced raw Avro, or JSON, or Protobuf) — a common source of "poison message" deserialization errors.
3. struct.unpack(">I", frame[1:5]) reads the four ID bytes big-endian. ">I" is the exact format Confluent uses; get the endianness wrong and you fetch the wrong schema (or a nonexistent one).
4. fetch_schema(42) is the registry lookup — real clients cache this aggressively, because fetching per message would add a network round-trip to every record. The cache is why a registry outage does not immediately stall consumers that already hold the schema.
5. Only after the ID resolves to schema_42 can schemaless_reader decode the positional payload. The whole point: those 5 framing bytes exist because Avro payloads are meaningless without the writer schema they reference.

Output.

frame: 000000002a0e477261636501...   (magic 00 | id 0000002a | avro payload)
decoded: {'id': 7, 'name': 'Grace'}

Rule of thumb. Memorize "magic byte, four-byte big-endian schema ID, then the Avro payload." When a consumer throws a deserialization error, check byte 0 first — a wrong magic byte means a mis-produced or cross-format message, not a schema problem.

Common beginner mistakes.

• Adding a field without a default. It is not backward compatible; a new reader cannot supply the missing value for old bytes. Always attach a default.
• Assuming Avro is self-describing. It is not — no field names on the wire. Ship the registry, or ship an object-container file with an embedded schema header.
• Renaming a field by editing the name. That drops the old field and adds a new one. Use aliases to rename compatibly.
• Reusing a subject across unrelated record types. The subject's compatibility history assumes one evolving type; mixing types corrupts the compatibility guarantee.
• Fetching the schema per message. Cache by schema ID; a per-message registry call adds a network hop to every record.

Streaming interview question on Avro schema evolution

A senior interviewer might ask: "Your orders-value subject is on BACKWARD compatibility. A producer team wants to add a discount_cents field and also rename total to total_cents. Walk me through which of those changes are safe, what the schema must look like, the deploy order, and how you'd verify it against the registry before shipping."

Solution Using defaults, aliases, and a registry compatibility check

# orders_v2.py — evolve the orders schema backward-compatibly
import io
import fastavro

writer_v1 = fastavro.parse_schema({
    "type": "record", "name": "Order", "namespace": "com.pipecode",
    "fields": [
        {"name": "id",    "type": "long"},
        {"name": "total", "type": "long"},          # cents, old name
    ],
})

# v2: add discount_cents WITH default; rename total -> total_cents via ALIAS
reader_v2 = fastavro.parse_schema({
    "type": "record", "name": "Order", "namespace": "com.pipecode",
    "fields": [
        {"name": "id",             "type": "long"},
        {"name": "total_cents",    "type": "long", "aliases": ["total"]},  # safe rename
        {"name": "discount_cents", "type": "long", "default": 0},          # safe add
    ],
})

# Old producer emits v1
buf = io.BytesIO()
fastavro.schemaless_writer(buf, writer_v1, {"id": 1001, "total": 4999})
buf.seek(0)

# New consumer resolves v1 bytes through v2: alias maps total->total_cents,
# discount_cents filled from default
record = fastavro.schemaless_reader(buf, writer_v1, reader_v2)
print(record)   # {'id': 1001, 'total_cents': 4999, 'discount_cents': 0}

# registry_check.sh — dry-run the v2 schema against the BACKWARD subject
curl -s -X POST \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "{\"schema\": $(jq -Rs . < order_v2.avsc)}" \
  "$SR_URL/compatibility/subjects/orders-value/versions/latest" | jq .
# -> {"is_compatible": true}   (register only if this is true)

Step-by-step trace.

Step	Change	Verdict	Mechanism
Add discount_cents	new field, default: 0	safe (backward)	reader supplies default for old bytes
Rename total→total_cents	via aliases: ["total"]	safe (backward)	resolution matches old name to new field
Deploy order	consumers first	correct	new readers already read old writers
Verify	POST /compatibility	is_compatible: true	registry confirms before registration

Adding discount_cents with a default and renaming total to total_cents via an alias are both backward compatible: the new reader resolves old v1 bytes by matching total→total_cents through the alias and back-filling discount_cents from its default. A bare rename (editing the name with no alias) would instead drop total and add an unmatched total_cents with no default — a break the registry's BACKWARD mode rejects.

Output:

Old v1 bytes	Field	Resolved value	Source
{id:1001, total:4999}	id	1001	matched by name
	total_cents	4999	matched via alias total
	discount_cents	0	filled from default

Why this works — concept by concept:

• Default-backed add — discount_cents with default: 0 is a backward-compatible addition because resolution has a value to supply when the old bytes lack the field. The default is the whole mechanism.
• Alias-backed rename — aliases: ["total"] tells resolution that a writer field named total should map to the reader field total_cents. This is how Avro renames without re-encoding a single historical byte.
• Deploy order = consumers first — the change is backward compatible, so new readers already understand old writers; consumers roll out first, then producers, with no window where anything breaks.
• Registry dry-run — POST /compatibility/subjects/orders-value/versions/latest returns is_compatible: true/false before registration. Wiring it into CI turns a schema break into a failed build, not a production incident.
• Cost — one alias annotation, one default value, one CI call. The eliminated cost is a coordinated multi-team deploy and a re-encode of historical data. O(1) per schema change; resolution is O(fields) per message, negligible against the network cost.

Streaming
Topic — streaming
Streaming Avro and schema-evolution problems

Practice →

ETL Topic — etl ETL problems on Avro ingestion and container files

Practice →

---

3. Protobuf — field numbers, varint wire format, proto3

Protobuf makes the field number the permanent contract — names never touch the wire, so renames are free and reuse is catastrophic

The mental model in one line: Protocol Buffers encode each present field as a (field_number << 3) | wire_type tag followed by a compact value — integers as base-128 varints, strings as length-delimited blobs — which means the field number is the entire on-wire identity of a field, field names are a compile-time convenience that never ship, unset fields cost zero bytes, and unknown fields are skipped-and-preserved by any decoder. This is why avro vs protobuf is not a "which is smaller" contest but a "which evolution model fits your org" contest: Avro reconciles by name against a registry, Protobuf reconciles by number against a compiled descriptor, and each buys a different set of safe changes.

The wire format, precisely.

• Tag = (field_number << 3) | wire_type. Each field on the wire starts with a varint tag. The low 3 bits are the wire type; the rest is the field number. Wire type 0 = varint (int32/int64/bool/enum), 1 = 64-bit fixed, 2 = length-delimited (string/bytes/embedded message/packed repeated), 5 = 32-bit fixed.
• Varints. Integers are base-128, little-endian, with the high bit of each byte as a continuation flag. Small numbers cost one byte; this is why field numbers 1–15 (1-byte tag) should be reserved for your hottest fields and 16–2047 (2-byte tag) for the rest.
• Zig-zag for signed. sint32/sint64 zig-zag-encode so small-magnitude negatives stay short. Plain int32 varint-encodes negatives as 10 bytes — a classic footgun.
• Length-delimited. A string is tag + length-varint + UTF-8 bytes. An embedded message is tag + length + the message's own tag/value stream — recursion all the way down.

proto3 semantics — presence, defaults, and the optional keyword.

• No required. proto3 removed required entirely; every field is optional in the sense that it can be absent. This was deliberate — required is un-evolvable (you can never remove a required field safely).
• Scalar defaults, no presence by default. In proto3, an unset scalar reads back as its zero value (0, "", false) and, by default, you cannot distinguish "set to zero" from "never set." That ambiguity bites when zero is a meaningful value.
• The optional keyword restores presence. Marking a proto3 field optional generates has_field() accessors and tracks explicit presence — the fix for "I need to know whether the client sent 0 or sent nothing."
• Repeated and packed. repeated fields concatenate; scalar repeated fields are packed by default in proto3 (one tag, then all values length-delimited) — a real space win over one-tag-per-element.

Evolution rules — the field number is forever.

• Renaming is free. Names never touch the wire, so total → total_cents is a source-only change; existing bytes decode identically.
• Adding is safe. A new field gets a new, never-used number; old decoders skip it as an unknown field and preserve it (so a proxy can re-serialize without data loss).
• Removing requires reserved. When you delete a field, reserved 4; and reserved "old_name"; fence off the number and name so no future field can reuse them. Reusing a number silently reinterprets old bytes — the worst class of Protobuf bug.
• Type changes are constrained. Some are wire-compatible (int32↔int64↔bool share wire type 0); most are not. Changing a field's type without changing its number is where subtle corruption lives.

Where Protobuf wins and loses.

• Wins. Partially self-describing (skip unknowns without the schema); tiny sparse records (unset = zero bytes); first-class gRPC/RPC story; polyglot codegen; presence via optional.
• Loses. Larger than Avro for fully-populated records (per-field tags); the compiled-descriptor build step; proto3's zero-value ambiguity; no registry-native name resolution (numbers, not names, are the contract).

Worked example — define a message, serialize, and inspect the bytes

Detailed explanation. The canonical Protobuf round-trip: write a .proto, compile it with protoc, then serialize and deserialize in Python while inspecting the raw tag/value bytes. Seeing 08 2a decode to "field 1, varint, value 42" is what makes the wire format click.

• Message. User { int64 id = 1; string name = 2; bool active = 3; }.
• Compile. protoc --python_out=. user.proto → user_pb2.py.
• Inspect. Serialize and read the hex.

Question. Serialize User(id=42, name="Ada", active=true) and decode the tag byte of each field by hand.

Input.

Field	Number	Wire type	Tag byte	Value bytes
id	1	0 (varint)	08	2a (42)
name	2	2 (len)	12	03 41 64 61
active	3	0 (varint)	18	01 (true)

Code.

// user.proto
syntax = "proto3";
package pipecode;

message User {
  int64  id     = 1;
  string name   = 2;
  bool   active = 3;
}

# proto_roundtrip.py  (after: protoc --python_out=. user.proto)
import user_pb2

u = user_pb2.User(id=42, name="Ada", active=True)
raw = u.SerializeToString()
print("bytes:", raw.hex(), "len:", len(raw))   # 082a1203416461 1801 -> 9 bytes

# Decode the tag of field 1 by hand: 0x08 -> (1 << 3) | 0
tag = raw[0]
print("field_number:", tag >> 3, "wire_type:", tag & 0x07)   # 1, 0 (varint)

back = user_pb2.User()
back.ParseFromString(raw)
print("decoded:", back.id, back.name, back.active)

Step-by-step explanation.

1. The .proto assigns numbers 1, 2, 3. Those numbers — not the names id, name, active — are what the wire format carries. Compiling with protoc produces user_pb2.py with a generated User class.
2. SerializeToString() walks the set fields in field-number order. Field 1 emits tag 0x08 = (1 << 3) | 0 (field 1, wire type 0 = varint) then the varint 0x2a = 42.
3. Field 2 (name) emits tag 0x12 = (2 << 3) | 2 (field 2, wire type 2 = length-delimited), then length 0x03, then the UTF-8 bytes 41 64 61 = "Ada".
4. Field 3 (active) emits tag 0x18 = (3 << 3) | 0, then 0x01 for true. Total 9 bytes — 3 more than Avro's 6 because each field pays a 1-byte tag, but that tag is what lets a decoder skip an unknown field.
5. tag >> 3 recovers the field number and tag & 0x07 the wire type — the exact arithmetic every Protobuf decoder runs. ParseFromString reverses the whole process using the compiled descriptor to map numbers back to typed fields.

Output.

bytes: 082a1203416461 1801 len: 9
field_number: 1 wire_type: 0
decoded: 42 Ada True

Rule of thumb. Learn to read a Protobuf tag byte in your head: tag >> 3 is the field number, tag & 7 is the wire type. Interviewers who ask "decode 0x12" are checking whether you actually understand the wire format or just call SerializeToString.

Worked example — decode a varint and a full message by hand

Detailed explanation. Varint decoding is the single most-asked Protobuf mechanics question. A varint stores an integer in base-128, little-endian, with each byte's high bit signalling "more bytes follow." Implement the decoder and walk a message.

• Continuation bit. High bit set (0x80) → another byte follows.
• Little-endian groups. The low 7 bits of each byte are the value, least-significant group first.
• Example. 0x96 0x01 → 0b0010110 0b0000001 reassembled = 150.

Question. Implement a varint reader and a minimal Protobuf field walker, then decode 08 96 01 12 03 41 64 61 (a message with id=150, name="Ada").

Input.

Bytes	Meaning
08	tag: field 1, varint
96 01	varint 150
12	tag: field 2, length-delimited
03 41 64 61	length 3 + "Ada"

Code.

# varint_decode.py — hand-roll a Protobuf field walker
def read_varint(data: bytes, pos: int) -> tuple[int, int]:
    result, shift = 0, 0
    while True:
        b = data[pos]; pos += 1
        result |= (b & 0x7F) << shift        # low 7 bits, little-endian
        if not (b & 0x80):                   # high bit clear -> last byte
            return result, pos
        shift += 7

def walk_message(data: bytes) -> list[tuple[int, int, object]]:
    pos, out = 0, []
    while pos < len(data):
        tag, pos = read_varint(data, pos)
        field_number, wire_type = tag >> 3, tag & 0x07
        if wire_type == 0:                       # varint
            value, pos = read_varint(data, pos)
        elif wire_type == 2:                     # length-delimited
            length, pos = read_varint(data, pos)
            value, pos = data[pos:pos + length], pos + length
        else:
            raise ValueError(f"wire type {wire_type} not handled")
        out.append((field_number, wire_type, value))
    return out

msg = bytes.fromhex("08 96 01 12 03 41 64 61".replace(" ", ""))
for fn, wt, val in walk_message(msg):
    print(f"field {fn} wire {wt}: {val!r}")

Step-by-step explanation.

1. read_varint accumulates 7 bits per byte, shifting each group left by a growing shift. For 96 01: first byte 0x96 has the continuation bit set → take low 7 bits 0b0010110 (22); second byte 0x01 clears the bit → take 0b0000001 (1) shifted by 7 → 1 << 7 = 128. Sum 128 + 22 = 150.
2. walk_message reads a tag varint, splits it into field number (tag >> 3) and wire type (tag & 7), then dispatches on wire type — exactly how a real parser skips fields it does not recognise.
3. Field 1 (08) is wire type 0, so the next varint (96 01 = 150) is its value — this is the id.
4. Field 2 (12) is wire type 2 (length-delimited), so read a length varint (03), then that many bytes (41 64 61 = "Ada") — this is the name.
5. The power of this walk: it decoded the message without the .proto. It could not name the fields or type them, but it read the structure and could skip an unknown field cleanly — the mechanism behind Protobuf's forward-skip resilience.

Output.

field 1 wire 0: 150
field 2 wire 2: b'Ada'

Rule of thumb. Master the varint continuation-bit trick — it is the mechanical heart of Protobuf. If you can decode 96 01 to 150 on a whiteboard, you can answer almost any Protobuf wire-format question.

Worked example — evolve a message safely with reserved and optional

Detailed explanation. Protobuf evolution lives and dies by the field number. Walk through the three moves — rename, add, remove — and the reserved/optional guardrails that keep them safe.

• Rename. Change the name in the .proto; the wire is unaffected.
• Add. New field, new number, optional if you need presence.
• Remove. Delete the field and reserved its number and name so nobody reuses them.

Question. Evolve User to rename name→full_name, add an optional string email = 4, and remove field 3 (active) safely.

Input.

Move	Change	Guardrail
rename	name → full_name (still field 2)	none needed (names off-wire)
add	optional string email = 4	optional for presence
remove	drop bool active = 3	reserved 3; reserved "active";

Code.

// user_v2.proto — evolved safely
syntax = "proto3";
package pipecode;

message User {
  reserved 3;                 // field 3 (active) removed — never reuse the number
  reserved "active";          // ...and never reuse the name

  int64  id        = 1;
  string full_name = 2;       // renamed from 'name' — SAME number 2, wire-identical
  optional string email = 4;  // NEW — 'optional' tracks explicit presence
}

# proto_evolution.py — old v1 bytes still decode under v2
import user_v2_pb2

# Bytes produced by the OLD schema: id=42, name="Ada", active=true
old_bytes = bytes.fromhex("082a1203416461 1801".replace(" ", ""))

u = user_v2_pb2.User()
u.ParseFromString(old_bytes)          # field 3 (active) now an UNKNOWN field
print("id:", u.id, "full_name:", u.full_name)   # 42 Ada  (renamed field reads fine)
print("has_email:", u.HasField("email"))        # False — optional presence works
# field 3 bytes are retained in u.UnknownFields(), not lost

Step-by-step explanation.

1. Renaming name→full_name keeps field number 2, so the old bytes 12 03 41 64 61 decode into full_name unchanged. Names are compile-time only; the wire never noticed.
2. Adding optional string email = 4 uses a brand-new number 4. Old bytes have nothing at field 4, so HasField("email") is False — the optional keyword is what makes that presence query legal in proto3.
3. Removing active (field 3) and adding reserved 3; reserved "active"; fences the number and name. When the v2 decoder hits the old 18 01 bytes (field 3), it treats them as an unknown field — skipped, but preserved in UnknownFields() so a re-serialize does not drop them.
4. The catastrophic alternative — deleting active and later adding optional int64 score = 3 reusing number 3 — would make the decoder read the old active=true bytes as score=1. Silent data corruption. reserved is the guardrail that makes this impossible.
5. Because every change here is number-safe, old producers and new consumers interoperate in any order — Protobuf's evolution model is permissive precisely because the number, not the name or position, is the contract.

Output.

id: 42 full_name: Ada
has_email: False

Rule of thumb. Every time you delete a Protobuf field, add reserved <number>; and reserved "<name>"; in the same commit. The reserved keyword is cheap insurance against the single worst Protobuf bug: a reused field number silently misreading old bytes.

Common beginner mistakes.

• Reusing a field number. The number is the permanent identity; reuse silently reinterprets old bytes. Always reserved a removed number.
• Using int32 for signed values. Negative int32 varint-encodes to 10 bytes. Use sint32/sint64 (zig-zag) for values that go negative.
• Relying on proto3 zero-value presence. An unset scalar reads as 0/""/false; you cannot tell "sent zero" from "unset" unless you mark the field optional.
• Packing hot fields into high numbers. Field numbers 1–15 get a 1-byte tag; put your most frequent fields there and leave 16+ for the rest.
• Forgetting the codegen step. Protobuf needs protoc compilation; there is no schemaless decode of a typed message without the descriptor.

Streaming interview question on Protobuf evolution and presence

A senior interviewer might ask: "A payments service uses proto3. A field int64 amount_cents = 5 exists, and the team needs to (a) know whether the client explicitly sent amount_cents = 0 versus never set it, and (b) add a currency field without breaking old consumers. Walk me through the schema changes, why proto3 makes (a) tricky, and how the wire format guarantees (b) is safe."

Solution Using optional for presence and a new field number for the additive change

// payment_v2.proto
syntax = "proto3";
package pipecode;

message Payment {
  int64  id             = 1;
  optional int64 amount_cents = 5;   // 'optional' -> HasField() distinguishes 0 from unset
  string currency       = 6;         // NEW field, new number 6 -> old readers skip it
  reserved 2, 3, 4;                  // numbers retired in earlier versions
}

# payment_presence.py  (after: protoc --python_out=. payment_v2.proto)
import payment_v2_pb2

# Case A: client explicitly sends amount_cents = 0
explicit_zero = payment_v2_pb2.Payment(id=1, amount_cents=0)
print(explicit_zero.HasField("amount_cents"))   # True  -> they DID send 0

# Case B: client never sets amount_cents
never_set = payment_v2_pb2.Payment(id=2)
print(never_set.HasField("amount_cents"))        # False -> genuinely unset

# Additive safety: OLD bytes (no 'currency') decode under v2 with currency = ""
old_bytes = payment_v2_pb2.Payment(id=3, amount_cents=999).SerializeToString()
v2 = payment_v2_pb2.Payment()
v2.ParseFromString(old_bytes)
print(v2.currency == "")                          # True -> default, no break

Step-by-step trace.

Step	Change	Effect
Mark amount_cents optional	proto3 optional keyword	HasField() distinguishes 0 from unset
Explicit zero	Payment(amount_cents=0)	HasField → True
Never set	Payment()	HasField → False
Add currency field 6	new, never-used number	old readers skip; new readers see ""
Retire old numbers	reserved 2, 3, 4	prevents catastrophic reuse

Marking amount_cents as optional restores explicit presence tracking that proto3 drops by default, so HasField("amount_cents") returns True for a sent zero and False for a genuinely unset field — the exact distinction a payments system needs to avoid treating "no amount" as "$0.00." Adding currency as field 6 is additive and safe: old producers emit no field-6 bytes, so old and new consumers read currency as the default "" with no break.

Output:

Scenario	HasField("amount_cents")	amount_cents value	currency
Explicit zero	True	0	""
Never set	False	0 (default)	""
Old v1 bytes	True	999	"" (default)

Why this works — concept by concept:

• proto3 optional presence — by default proto3 collapses "unset" and "zero" into the same read-back value; the optional keyword generates a presence bit and HasField() accessor, which is the only correct way to model a nullable scalar where zero is meaningful.
• Additive field by new number — currency = 6 uses a number no prior version used, so old encoders simply omit it and every decoder falls back to the scalar default. Additivity is guaranteed by the wire format, not by a registry.
• Reserved retired numbers — reserved 2, 3, 4 fences off numbers used by dead fields so a future addition cannot reuse one and misread historical bytes.
• Number-as-contract — because names never touch the wire, none of these changes require re-encoding data or a coordinated deploy; old and new peers interoperate in any order.
• Cost — one optional keyword (a presence bit per marked field), one new field number, one reserved line, and a protoc recompile. The eliminated cost is a class of "0 vs null" logic bugs and a coordinated migration. O(1) per schema change; decode remains O(fields).

Streaming
Topic — streaming
Streaming Protobuf and wire-format problems

Practice →

Design Topic — design Design problems on typed event contracts

Practice →

---

4. JSON Schema — human-readable validation vs serialization

JSON Schema validates a self-describing payload — it never shrinks the wire, and its evolution guarantees are weaker, but debuggability and zero-decode-dependency are real wins

The mental model in one line: JSON Schema is a validation language layered over plain JSON — the payload is fully self-describing text that any parser decodes without a schema, and the schema's job is only to assert the decoded object obeys a contract (type, required, additionalProperties) — so JSON Schema buys you human-readable, zero-decode-dependency messages and a real registry-enforced contract, at the cost of the largest wire size of the three and evolution rules that are looser and easier to get subtly wrong. The avro vs protobuf debate is fundamentally about binary trade-offs; JSON Schema is the third option you reach for when a human reading the raw message in a debugger, or a consumer that must work with no schema fetch, is worth more than the wire bytes.

Validation is not serialization — the distinction that trips people up.

• Serialization is done by JSON itself. json.dumps / json.loads turn objects into text and back. No schema is involved; JSON is self-describing.
• Validation is what the schema does. A JSON Schema document asserts that a decoded object has the right types, the required keys, and (optionally) no extra keys (additionalProperties: false). It runs after decode.
• Enforcement is opt-in unless you wire it in. Nothing forces validation. A consumer that skips validate() will happily process {"id": "not-a-number"} until arithmetic on id explodes downstream. This is JSON's freedom and its footgun.

The schema keywords that carry the contract.

• type and properties. Declare each field's JSON type (integer, string, boolean, object, array) and shape.
• required. The list of keys that must be present. Adding to required is the classic breaking change.
• additionalProperties. true (default) allows unknown keys through; false rejects them. This one keyword flips your forward-compatibility story.
• $schema and $id. Declare the dialect (Draft 7, 2020-12) and identity so validators and registries agree on semantics.

Evolution rules — looser and more error-prone than Avro/Protobuf.

• Add an optional field (not in required, additionalProperties permissive). Backward and forward compatible — old readers ignore it, new readers accept its absence.
• Add a required field. Breaks backward compatibility: old messages lack it, new readers reject them. The single most common JSON Schema break.
• additionalProperties: false inverts the story. With strict mode, a new field in the data fails an old schema's validation — so adding a field is no longer forward compatible. Confluent's JSON Schema compatibility rules hinge on this keyword.
• No positional decode, no field-number contract. Renames are breaks (the key is the identity, and it is in the payload). There is no alias mechanism and no reserved-number safety net.

Where JSON Schema wins and loses.

• Wins. Human-readable in any log, debugger, or kafka-console-consumer; zero decode dependency (no registry fetch to read the bytes); ubiquitous tooling; the natural fit for web APIs, config, and low-volume, high-debuggability streams.
• Loses. The largest wire size (repeated field names, decimal-string numbers); weakest and most footgun-prone evolution rules; validation is a separate, skippable step; no compact binary path.

Worked example — define a schema and validate a payload

Detailed explanation. The canonical JSON Schema flow: serialize with plain JSON, then validate the decoded object against a Draft 2020-12 schema. Watch validation reject a bad type that Avro/Protobuf would have refused to even encode.

• Schema. id: integer, name: string, active: boolean; required: [id, name]; additionalProperties: false.
• Serialize. json.dumps — no schema needed.
• Validate. jsonschema.validate — raises on violation.

Question. Validate {"id": 42, "name": "Ada", "active": true} (valid) and {"id": "oops", "name": "Ada"} (invalid type) against the schema.

Input.

Field	Schema type	Required?	Valid payload	Invalid payload
id	integer	yes	42	"oops" (string)
name	string	yes	"Ada"	"Ada"
active	boolean	no	true	(omitted)

Code.

# json_schema_validate.py   (pip install jsonschema)
import json
from jsonschema import Draft202012Validator, ValidationError

schema = {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$id": "https://pipecode.ai/schemas/user.json",
    "type": "object",
    "properties": {
        "id":     {"type": "integer"},
        "name":   {"type": "string"},
        "active": {"type": "boolean"},
    },
    "required": ["id", "name"],
    "additionalProperties": False,
}
validator = Draft202012Validator(schema)

valid   = {"id": 42, "name": "Ada", "active": True}
invalid = {"id": "oops", "name": "Ada"}

payload = json.dumps(valid, separators=(",", ":")).encode()   # 36 bytes, self-describing
print("wire bytes:", len(payload))

validator.validate(valid)        # passes silently
try:
    validator.validate(invalid)  # raises: 'oops' is not of type 'integer'
except ValidationError as e:
    print("rejected:", e.message)

Step-by-step explanation.

1. json.dumps(valid) produces 36 bytes of self-describing text — the field names id, name, active are in the payload, which is why any consumer decodes it with no schema at all.
2. The schema is a plain dict with $schema (the dialect) and $id (its identity). Draft202012Validator compiles it once; reuse the compiled validator per message rather than calling validate() fresh each time.
3. validator.validate(valid) passes silently — id is an integer, name a string, active a boolean, and no extra keys violate additionalProperties: false.
4. validator.validate(invalid) raises ValidationError because "oops" is a string, not the declared integer. This is the enforcement Avro/Protobuf get for free at encode time but JSON only gets if you choose to validate.
5. The load-bearing risk: if a consumer skips validate(), {"id": "oops"} sails straight through json.loads into application code. JSON's contract is only as strong as the discipline to run validation on every message.

Output.

wire bytes: 36
rejected: 'oops' is not of type 'integer'

Rule of thumb. Treat JSON Schema validation as mandatory, not optional — wire it into the deserializer so no message reaches application code unvalidated. JSON's self-describing convenience is exactly what lets bad data through if you skip the check.

Worked example — the required-field break and additionalProperties

Detailed explanation. The two JSON Schema evolution footguns are required and additionalProperties. Adding a required field breaks old data; strict additionalProperties: false breaks new data under an old schema. Walk both.

• Add optional field. Safe both directions.
• Add required field. Old messages lack it → new schema rejects them (backward break).
• additionalProperties: false. New field in data → old schema rejects it (forward break).

Question. Show that adding email to required rejects an old message, and that additionalProperties: false rejects a message carrying a new phone field.

Input.

Change	Old data / old schema	Verdict
add optional email	old data lacks it	pass (not required)
add required email	old data lacks it	FAIL (backward break)
new phone in data, additionalProperties:false	old schema	FAIL (forward break)

Code.

# json_schema_evolution.py
from jsonschema import Draft202012Validator, ValidationError

old_message = {"id": 1, "name": "Grace"}                       # produced before 'email'

# v2a: email added but OPTIONAL -> old message still valid
v2a = Draft202012Validator({
    "type": "object",
    "properties": {"id": {"type": "integer"}, "name": {"type": "string"},
                   "email": {"type": "string"}},
    "required": ["id", "name"],
})
v2a.validate(old_message)   # passes: email not required

# v2b: email added to REQUIRED -> old message now invalid
v2b = Draft202012Validator({
    "type": "object",
    "properties": {"id": {"type": "integer"}, "name": {"type": "string"},
                   "email": {"type": "string"}},
    "required": ["id", "name", "email"],
})
try:
    v2b.validate(old_message)     # raises: 'email' is a required property
except ValidationError as e:
    print("backward break:", e.message)

# Strict schema rejects an unforeseen new field
strict = Draft202012Validator({
    "type": "object",
    "properties": {"id": {"type": "integer"}, "name": {"type": "string"}},
    "required": ["id", "name"], "additionalProperties": False,
})
try:
    strict.validate({"id": 2, "name": "Ada", "phone": "555-0100"})  # new field
except ValidationError as e:
    print("forward break:", e.message)

Step-by-step explanation.

1. v2a adds email to properties but not to required, so the old message {"id": 1, "name": "Grace"} still validates — the field is optional and its absence is legal. This is the safe additive change.
2. v2b adds email to required. Now the old message fails validation with "'email' is a required property" — a backward break, because a reader on the new schema cannot accept old data. This is the most common JSON Schema evolution mistake.
3. The strict schema sets additionalProperties: false. A message carrying a new phone field the schema never declared is rejected — a forward break, because old readers refuse new producers' extra fields.
4. The asymmetry is the lesson: with additionalProperties: true (the default), adding a field is forward compatible (old readers ignore it); with false, the same addition is a forward break. One keyword flips the entire compatibility direction.
5. Confluent's JSON Schema compatibility checker encodes exactly these rules — it inspects required and additionalProperties to decide whether a new schema version is BACKWARD/FORWARD/FULL compatible, and blocks registration otherwise.

Output.

backward break: 'email' is a required property
forward break: Additional properties are not allowed ('phone' was unexpected)

Rule of thumb. Add new JSON Schema fields as optional and keep additionalProperties permissive on consumers if you want forward compatibility. Only two moves reliably break JSON Schema evolution — adding to required and flipping additionalProperties to false — so audit every schema change for those two.

Worked example — JSON Schema on Kafka via the registry

Detailed explanation. Confluent supports JSON Schema in the same registry as Avro and Protobuf, with the same 5-byte framing and the same compatibility modes. The difference is that the framed payload is JSON text, not binary. Walk the framing and the trade-off.

• Same framing. Magic byte 0x00 + 4-byte schema ID + JSON bytes.
• Same modes. BACKWARD/FORWARD/FULL/*_TRANSITIVE apply, interpreted via required/additionalProperties.
• The trade-off. You get registry enforcement and readability, but you keep JSON's wire size.

Question. Show the JSON Schema serializer framing and confirm the payload after the 5 framing bytes is still human-readable JSON.

Input.

Byte range	Meaning
[0]	magic 0x00
[1:5]	schema ID
[5:]	JSON text (readable!)

Code.

# json_schema_kafka.py — framing mirrors Avro/Protobuf, payload stays readable
import json
import struct

def serialize_json_schema(record: dict, schema_id: int) -> bytes:
    body = json.dumps(record, separators=(",", ":")).encode("utf-8")
    return b"\x00" + struct.pack(">I", schema_id) + body     # magic + id + JSON

def deserialize_json_schema(frame: bytes) -> tuple[int, dict]:
    assert frame[0] == 0, "not a Confluent-framed message"
    schema_id = struct.unpack(">I", frame[1:5])[0]
    return schema_id, json.loads(frame[5:])                   # no registry fetch to DECODE

frame = serialize_json_schema({"id": 42, "name": "Ada"}, schema_id=17)
print("frame:", frame.hex())
print("payload is readable:", frame[5:].decode())            # {"id":42,"name":"Ada"}
print("decoded:", deserialize_json_schema(frame))

Step-by-step explanation.

1. serialize_json_schema prepends the exact Confluent framing — magic 0x00, big-endian schema ID, then the body. The only difference from Avro is that the body is UTF-8 JSON text instead of a positional binary stream.
2. The registry still governs the schema: the producer registered the JSON Schema under orders-value, got ID 17, and stamped it into bytes 1–4 so consumers can fetch the schema for validation.
3. deserialize_json_schema reads the ID but then calls json.loads(frame[5:]) — decoding needs no registry fetch because JSON is self-describing. The schema fetch is only needed to validate, and even that is optional.
4. frame[5:].decode() prints readable {"id":42,"name":"Ada"} — the debuggability win. An engineer tailing the topic with kafka-console-consumer sees fields, not hex, at the cost of the wire size those field names imply.
5. This is the JSON Schema sweet spot on Kafka: registry-enforced contracts and compatibility CI (same as Avro/Protobuf) plus eyeball-debuggable messages — for streams where that combination beats the binary formats' smaller footprint.

Output.

frame: 0000000011 7b226964223a34322c226e616d65223a22416461227d
payload is readable: {"id":42,"name":"Ada"}
decoded: (17, {'id': 42, 'name': 'Ada'})

Rule of thumb. Reach for JSON Schema on Kafka when human-debuggability and zero-decode-dependency outweigh wire size — think low-volume domain events, audit trails, and integration boundaries with external partners. You keep registry enforcement; you pay in bytes.

Common beginner mistakes.

• Skipping validation. JSON decodes without a schema, so nothing enforces the contract unless you call validate() on every message. Wire it into the deserializer.
• Adding a field to required. That is a backward break — old messages lack it. Add new fields as optional.
• Not thinking about additionalProperties. false makes adding a field a forward break; true makes it safe. Choose deliberately.
• Treating renames as free. The JSON key is the identity and it is in the payload; a rename is a break with no alias escape hatch.
• Assuming JSON is "good enough" at scale. The repeated-field-name tax is 3–8× versus binary; measure it before committing high-volume topics to JSON.

JSON interview question on JSON Schema evolution

A senior interviewer might ask: "An external partner consumes your events topic as JSON and can't redeploy on your schedule. You need to add a region field. Walk me through whether to make it required, what additionalProperties setting keeps the partner working, which compatibility mode the registry should enforce, and how you'd validate a message on the way in."

Solution Using an optional field, permissive additionalProperties, and FORWARD compatibility

# events_evolution.py — add 'region' without breaking a partner who can't redeploy
from jsonschema import Draft202012Validator, ValidationError

# v2 schema: region is OPTIONAL, additionalProperties stays permissive
v2_schema = {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "event_id": {"type": "string"},
        "ts":       {"type": "integer"},
        "region":   {"type": "string"},     # NEW, optional
    },
    "required": ["event_id", "ts"],          # region NOT required
    "additionalProperties": True,            # partner's unknown fields tolerated
}
validator = Draft202012Validator(v2_schema)

# Partner (old producer) emits a message WITHOUT region
old_partner_msg = {"event_id": "e-1", "ts": 1_722_600_000}
validator.validate(old_partner_msg)          # passes: region optional

# Our new producer emits WITH region; partner's old reader ignores it
new_msg = {"event_id": "e-2", "ts": 1_722_600_100, "region": "eu-west-1"}
validator.validate(new_msg)                  # passes

def guarded_decode(raw: bytes) -> dict:
    import json
    obj = json.loads(raw)                     # decode (self-describing)
    validator.validate(obj)                   # THEN validate on the way in
    return obj

# set the subject to FORWARD so producers can add fields ahead of consumers
curl -s -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FORWARD_TRANSITIVE"}' \
  "$SR_URL/config/events-value" | jq .

Step-by-step trace.

Step	Decision	Reason
region optionality	NOT in required	old partner messages must still validate
additionalProperties	true	tolerate fields the partner sends that we don't model
Compatibility mode	FORWARD_TRANSITIVE	producers add fields before consumers upgrade
Validation point	on decode (guarded_decode)	no message reaches app code unvalidated

Making region optional keeps the partner's old messages valid, and additionalProperties: true means neither side rejects a field the other doesn't know. Because our producer ships the new field before the partner upgrades, the safe direction is forward compatibility — old readers (the partner) must read new writers (us) — so the subject is set to FORWARD_TRANSITIVE, which the registry enforces against every prior version.

Output:

Message	Has region?	Valid under v2?	Partner (old reader) handles?
old partner msg	no	yes (optional)	yes
our new msg	yes	yes	yes (ignores unknown field)
malformed (ts:"x")	—	no (type error)	rejected at guarded_decode

Why this works — concept by concept:

• Optional new field — leaving region out of required makes its absence legal, so messages produced before the change still validate. Required-field additions are the canonical backward break; optional additions avoid it.
• Permissive additionalProperties — true lets each side carry fields the other has not modelled, which is essential when an external partner evolves on its own timeline and may add fields you never declared.
• FORWARD compatibility — because we (producers) ship the field before the partner (consumers) upgrades, the guarantee we need is old-reads-new. FORWARD_TRANSITIVE checks that against all prior versions, not just the immediately preceding one.
• Validate on the way in — guarded_decode decodes then validates so malformed or wrong-typed messages are caught at the boundary, not deep in business logic where the stack trace is useless.
• Cost — one optional field, one config change, one validation call per message. The eliminated cost is a coordinated deploy with a partner you do not control. Validation is O(fields) per message — cheap next to the network cost, and the price of a real contract over self-describing data.

JSON
Topic — json
JSON Schema validation and payload-contract problems

Practice →

ETL Topic — etl ETL problems on JSON ingestion and schema drift

Practice →

---

5. Head-to-head — compatibility, registry, decision matrix

The registry's compatibility mode is the real contract — pick the format for the axis you're bound by, then let backward/forward/full govern every deploy

The mental model in one line: once you strip away the syntax, avro vs protobuf vs JSON Schema reduces to two orthogonal decisions — which format (chosen by the axis your workload is bound by: wire size → Avro, typed contracts → Protobuf, debuggability → JSON Schema) and which compatibility mode the schema registry enforces (BACKWARD for consumer-first deploys, FORWARD for producer-first, FULL for any-order, each _TRANSITIVE variant checking all prior versions) — and the compatibility mode, not the format, is what actually keeps a fleet of independently-deployed producers and consumers from breaking each other. Every senior engineer eventually learns that the format is a one-time decision and the compatibility mode is a decision you live with on every pull request.

The four compatibility modes (and their transitive twins).

• BACKWARD (the default). A new schema can read data written by the previous schema. Upgrade consumers first. Allowed: delete fields, add optional/defaulted fields. This is the safe default because you usually control consumer rollout more tightly than producer rollout.
• FORWARD. Data written by the new schema can be read by the previous schema. Upgrade producers first. Allowed: add fields, delete optional/defaulted fields.
• FULL. Both directions hold — deploy in any order. Allowed: add or delete only optional/defaulted fields. The most restrictive and the safest.
• *_TRANSITIVE. Each mode has a transitive variant that checks the new schema against all previous versions, not just the immediately preceding one. Non-transitive checks only version N vs N−1; transitive checks N vs {N−1 … 1}. For long-lived topics with old data still in retention, transitive is the honest choice.

How each format maps its safe changes onto the modes.

• Avro. BACKWARD = add-with-default / delete; FORWARD = add / delete-with-default; FULL = add-or-delete-with-default. Defaults and aliases are the levers.
• Protobuf. More permissive because field numbers carry identity — adding fields is broadly compatible in both directions, and the registry's Protobuf checker allows more than Avro's. Removing a field is compatible if you reserved the number.
• JSON Schema. Governed by required and additionalProperties. BACKWARD roughly = don't add required, don't tighten; FORWARD roughly = don't remove required, keep additionalProperties permissive.

Registry integration — one registry, three serdes.

• One registry, one framing. Confluent Schema Registry stores Avro, Protobuf, and JSON Schema, all behind the same magic-byte + schema-ID framing and the same compatibility API.
• Subject naming strategy. TopicNameStrategy (default) → <topic>-value; RecordNameStrategy and TopicRecordNameStrategy let multiple record types share a topic. The strategy decides what the compatibility history is scoped to.
• The CI gate is format-agnostic. POST /compatibility/subjects/<subject>/versions/latest returns is_compatible for any of the three formats — the same shift-left check regardless of which serde you picked.

The decision, distilled.

• Pick Avro when wire size dominates and you already run the Kafka/Confluent ecosystem: high-volume telemetry, data-lake landing, resolution-based evolution.
• Pick Protobuf when you want strong typing, gRPC reuse, and number-based evolution: typed domain events, polyglot microservices, contracts that also serve RPC.
• Pick JSON Schema when debuggability and zero-decode-dependency beat bytes: low-volume events, external-partner boundaries, audit trails, config streams.
• Then pick the mode by deploy order: consumers-first → BACKWARD, producers-first → FORWARD, any-order → FULL; add _TRANSITIVE whenever old data is still readable.

Worked example — the three-format decision matrix

Detailed explanation. The single artifact every serialization interview converges on is a compact matrix scoring the three formats across the axes. Build it and use it to place three concrete workloads.

• Axes. Schema location, wire size, evolution model, self-describing, tooling fit.
• Workloads. High-volume clickstream, typed order events, external-partner feed.

Question. Score Avro, Protobuf, and JSON Schema across five axes and assign each of the three workloads its format.

Input.

Axis	Avro	Protobuf	JSON Schema
Schema location	registry (mandatory)	compiled descriptor	in the payload
Wire size (full record)	smallest	small	largest
Wire size (sparse record)	small	smallest	largest
Evolution model	name + default + alias	field number + reserved	required + additionalProperties
Self-describing	no	partial (numbers)	yes

Code.

# decision_matrix.py — score formats and place workloads
FORMATS = {
    "avro":     {"wire_full": 3, "wire_sparse": 2, "typed": 2, "debuggable": 0, "registry": 3},
    "protobuf": {"wire_full": 2, "wire_sparse": 3, "typed": 3, "debuggable": 0, "registry": 2},
    "json":     {"wire_full": 0, "wire_sparse": 0, "typed": 1, "debuggable": 3, "registry": 2},
}  # higher = better on that axis

def place(volume: str, needs_typing: bool, human_read: bool) -> str:
    if human_read:
        return "json"                        # debuggability dominates
    if needs_typing:
        return "protobuf"                    # typed contracts / gRPC
    if volume == "high":
        return "avro"                        # wire size at scale
    return "avro"                            # ecosystem default

print("clickstream   ->", place("high", False, False))   # avro
print("order events  ->", place("low",  True,  False))   # protobuf
print("partner feed  ->", place("low",  False, True))    # json

Step-by-step explanation.

1. The matrix scores each format per axis (higher is better). Avro tops wire_full and registry; Protobuf tops wire_sparse and typed; JSON tops debuggable. No format wins every axis — the placement function encodes which axis the workload is bound by.
2. Clickstream is high-volume and not human-read, so place returns avro — the wire-size axis dominates and Avro's fully-populated-record compactness plus Confluent-native tooling win.
3. Order events need typed contracts (and often gRPC reuse across services), so place returns protobuf — the typing and number-based evolution axis wins even though volume is low.
4. The partner feed is human-read at an external boundary, so place returns json — debuggability and zero-decode-dependency beat the wire tax at low volume.
5. The lesson is that "which format is best" is the wrong question. "Which axis is this workload bound by" produces a defensible, per-workload answer — the exact framing a senior interviewer is listening for.

Output.

Workload	Bound by axis	Format
clickstream	wire size at scale	Avro
order events	typed contracts / gRPC	Protobuf
partner feed	debuggability	JSON Schema

Rule of thumb. Do not adopt one format company-wide by decree. Score the workload against the axes, name the axis it is bound by, and pick the format that wins that axis — then standardise the registry and compatibility discipline company-wide instead.

Worked example — choose the compatibility mode from the deploy order

Detailed explanation. The compatibility mode is chosen by one question: who upgrades first? Map the three deploy realities to the three modes and show why transitive matters for long-lived topics.

• Consumers upgrade first. BACKWARD — new readers must read old writers.
• Producers upgrade first. FORWARD — old readers must read new writers.
• Uncoordinated / any order. FULL — both must hold.

Question. For three deploy scenarios, pick the compatibility mode and state whether it must be transitive.

Input.

Scenario	Who upgrades first	Old data in retention?	Mode
Central platform ships consumers	consumers	yes	BACKWARD_TRANSITIVE
Producer team ships a new field first	producers	no	FORWARD
Independent teams, any order	either	yes	FULL_TRANSITIVE

Code.

# compat_mode.py — pick the registry compatibility mode
def pick_mode(consumers_first: bool, producers_first: bool, old_data_retained: bool) -> str:
    if consumers_first and not producers_first:
        base = "BACKWARD"
    elif producers_first and not consumers_first:
        base = "FORWARD"
    else:                                    # any order / uncoordinated
        base = "FULL"
    return base + ("_TRANSITIVE" if old_data_retained else "")

print(pick_mode(True,  False, True))    # BACKWARD_TRANSITIVE
print(pick_mode(False, True,  False))   # FORWARD
print(pick_mode(True,  True,  True))    # FULL_TRANSITIVE

# apply the chosen mode to the subject
curl -s -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FULL_TRANSITIVE"}' \
  "$SR_URL/config/orders-value" | jq .

Step-by-step explanation.

1. pick_mode maps the deploy reality to the base mode: consumers-first → BACKWARD, producers-first → FORWARD, any-order → FULL. The mode is a statement about ordering guarantees, not about the data.
2. The _TRANSITIVE suffix is added whenever old data is still in retention. Non-transitive BACKWARD only checks version N against N−1; if a message written by version N−5 is still on the topic (or in tiered storage), only BACKWARD_TRANSITIVE guarantees the new reader can read it too.
3. Scenario 1 (central platform, consumers-first, long retention) → BACKWARD_TRANSITIVE. The platform team upgrades consumers first and must read years of retained data, so transitive is mandatory.
4. Scenario 2 (producer ships a field first, short retention) → FORWARD. Producers lead, old consumers must tolerate the new field, and with no long-retained old data the non-transitive check suffices.
5. Scenario 3 (independent teams) → FULL_TRANSITIVE, the strictest. Any deploy order is possible and old data persists, so every change must be compatible in both directions against every prior version. The PUT /config/<subject> applies it and every future registration is checked against it.

Output.

BACKWARD_TRANSITIVE
FORWARD
FULL_TRANSITIVE

Rule of thumb. Choose the compatibility mode from a single question — "who deploys first?" — and add _TRANSITIVE the moment old data is still readable. FULL_TRANSITIVE is the correct default for topics consumed by teams you do not coordinate with; it is stricter and slower but it never strands a deploy.

Worked example — migrate a topic from JSON to Avro without downtime

Detailed explanation. Format migrations are the hardest evolution move because you cannot mix two formats in one partition. The dual-topic pattern migrates safely: dual-write both formats, migrate consumers, then retire the old topic. Walk the four phases.

• Phase 1. Producers dual-write orders-json and orders-avro.
• Phase 2. Migrate consumers one at a time to orders-avro.
• Phase 3. Confirm no consumer reads orders-json.
• Phase 4. Stop the JSON write; retire the topic.

Question. Design the dual-write migration from a JSON topic to an Avro topic and state how you know each phase is complete.

Input.

Phase	Producers	Consumers	Done when
1 dual-write	write both	read JSON	Avro topic has full traffic
2 migrate	write both	move to Avro	last consumer cut over
3 verify	write both	read Avro	JSON topic has zero consumer lag movement
4 retire	write Avro only	read Avro	JSON topic deleted

Code.

# dual_write_migration.py — phase 1: emit the same event to both topics
import io, json, struct, fastavro

avro_schema = fastavro.parse_schema({
    "type": "record", "name": "Order", "namespace": "com.pipecode",
    "fields": [{"name": "id", "type": "long"}, {"name": "total_cents", "type": "long"}],
})

def produce_both(producer, event: dict, avro_schema_id: int) -> None:
    # JSON topic (legacy consumers)
    producer.produce("orders-json",
                     value=json.dumps(event, separators=(",", ":")).encode())

    # Avro topic (new consumers) — Confluent framing
    buf = io.BytesIO()
    fastavro.schemaless_writer(buf, avro_schema, event)
    framed = b"\x00" + struct.pack(">I", avro_schema_id) + buf.getvalue()
    producer.produce("orders-avro", value=framed)

    producer.flush()

# Consumers migrate independently; when 'orders-json' consumer-group lag stops
# advancing (no active readers), phase 3 is complete and JSON write can stop.

Step-by-step explanation.

1. Phase 1 dual-writes: produce_both emits every event as JSON to orders-json and as Confluent-framed Avro to orders-avro. Both topics now carry the full stream, so a consumer can read from either.
2. You cannot transcode in place because a single partition cannot hold two formats without a discriminator — mixing raw JSON and magic-byte Avro bytes would make deserialization ambiguous. The separate topic is what makes the migration atomic per consumer.
3. Phase 2 migrates consumers one at a time to orders-avro. Each consumer group cuts over independently; a bug in one consumer's Avro path never affects the others still on JSON.
4. Phase 3 verifies completeness by watching orders-json's consumer-group offsets: when no group's offset advances, no live consumer reads JSON, and it is safe to stop producing it.
5. Phase 4 drops the JSON write and, after the retention window, deletes orders-json. The migration completed with zero downtime and no coordinated big-bang cutover — the only safe way to change a topic's wire format under a running fleet.

Output.

Phase	orders-json traffic	orders-avro traffic	Signal to advance
1 dual-write	full	full	Avro consumers validated
2 migrate	full	full	each consumer group cut over
3 verify	full (unread)	full	JSON group offsets frozen
4 retire	stopped	full	JSON topic deleted after retention

Rule of thumb. Never transcode a topic in place. Dual-write to a second topic, migrate consumers one group at a time, verify the old topic has no live readers via frozen consumer offsets, then retire it. Format migration is a four-phase dual-write, not a flag flip.

Common beginner mistakes.

• Confusing format choice with compatibility mode. They are orthogonal — format is a one-time decision, the mode governs every deploy. Decide both, explicitly.
• Using non-transitive compatibility with long retention. Non-transitive only checks N vs N−1; old retained data can still break. Use _TRANSITIVE when old data is readable.
• Assuming one format fits the whole company. Different workloads are bound by different axes; standardise the registry discipline, not the format.
• Transcoding a topic in place. One partition cannot mix two wire formats safely. Migrate via a dual-written second topic.
• Setting compatibility to NONE to "unblock" a change. That disables the only guardrail; the break just moves to production. Fix the schema instead.

Streaming interview question on format and compatibility selection

A senior interviewer might ask: "You own a Kafka platform with three topics — a 400k-events/sec clickstream, typed order events shared with gRPC services, and a low-volume feed to an external partner who reads raw messages. Producers and consumers are owned by different teams. Choose a serialization format per topic, choose a compatibility mode per topic, and justify each against the axes and the deploy realities."

Solution Using per-topic formats and per-topic compatibility modes

# platform_policy.py — codify the per-topic format + compatibility decision
from dataclasses import dataclass

@dataclass
class Topic:
    name: str
    events_per_sec: int
    typed_contract: bool       # shared with gRPC / strong typing needed?
    human_read: bool           # external partner reads raw messages?
    independent_deploys: bool  # producers/consumers deploy separately?
    old_data_retained: bool

def policy(t: Topic) -> tuple[str, str]:
    # format: pick the axis the topic is bound by
    if t.human_read:
        fmt = "json-schema"
    elif t.typed_contract:
        fmt = "protobuf"
    elif t.events_per_sec > 100_000:
        fmt = "avro"
    else:
        fmt = "avro"
    # compatibility: independent deploys -> FULL; add TRANSITIVE if old data lives
    mode = "FULL" if t.independent_deploys else "BACKWARD"
    if t.old_data_retained:
        mode += "_TRANSITIVE"
    return fmt, mode

topics = [
    Topic("clickstream",  400_000, False, False, True, True),
    Topic("orders",         3_000, True,  False, True, True),
    Topic("partner-feed",      40, False, True,  True, False),
]
for t in topics:
    fmt, mode = policy(t)
    print(f"{t.name:13} -> {fmt:11} | {mode}")

# apply each topic's compatibility mode to its subject
for sub in clickstream-value orders-value partner-feed-value; do
  curl -s -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
    --data '{"compatibility": "FULL_TRANSITIVE"}' "$SR_URL/config/$sub" >/dev/null
done

Step-by-step trace.

Topic	Bound by axis	Format	Deploy reality	Mode
clickstream	wire size (400k/s)	Avro	independent + retained	FULL_TRANSITIVE
orders	typed / gRPC	Protobuf	independent + retained	FULL_TRANSITIVE
partner-feed	debuggability	JSON Schema	independent, short retention	FULL

The clickstream is wire-size-bound at 400k/s, so Avro's compact fully-populated records win; the order events are typed and shared with gRPC services, so Protobuf's number-based contracts win; the partner feed is human-read at an external boundary, so JSON Schema's debuggability wins. Because every topic is consumed by teams that deploy independently, all three get FULL compatibility, and the two with long-retained data get FULL_TRANSITIVE so a new schema is checked against every historical version still readable.

Output:

Topic	Format	Compatibility	Justification
clickstream	Avro	FULL_TRANSITIVE	6× smaller than JSON at 400k/s
orders	Protobuf	FULL_TRANSITIVE	typed contracts + gRPC reuse
partner-feed	JSON Schema	FULL	partner reads raw messages

Why this works — concept by concept:

• Per-topic format — each topic is bound by a different axis, so a single company-wide format would be wrong for two of the three. The policy function names the binding axis and picks the format that wins it.
• FULL for independent deploys — when no team controls the deploy order, only both-directions compatibility guarantees that any ordering of producer/consumer rollouts is safe. Backward-only would strand producers that ship ahead.
• TRANSITIVE for retained data — long retention means old-version messages are still readable, so the new schema must be checked against all prior versions, not just the last one.
• Registry as the single enforcement plane — one registry stores all three formats behind one compatibility API, so the CI gate and the deploy guarantees are uniform even though the wire formats differ.
• Cost — one registry, three serdes, one PUT /config per subject, and a CI compatibility check per change. The eliminated cost is three separate ad-hoc contract mechanisms and the cross-team coordination they would need. O(1) enforcement per schema change per topic, independent of fleet size.

Streaming
Topic — streaming
Streaming compatibility-mode and registry problems

Practice →

Design
Topic — design
Design problems on schema-registry topology

Practice →

---

Cheat sheet — serialization & schema evolution recipes

• Which format when. Avro is the default when wire size dominates and you run Kafka/Confluent — smallest fully-populated records, resolution-based evolution, registry-native. Protobuf when you want strong typing, gRPC reuse, and number-based evolution — typed domain events, polyglot services, presence via optional. JSON Schema when human-debuggability and zero-decode-dependency beat bytes — low-volume events, external-partner feeds, audit trails. Never decree one format company-wide; pick per workload by the axis it is bound by, and standardise the registry discipline instead.
• The three axes. Schema location: Avro = registry-mandatory (no schema, no decode); Protobuf = compiled descriptor (skip unknowns without it); JSON Schema = in the payload (decode needs nothing). Wire size: Avro smallest for full records, Protobuf smallest for sparse records, JSON largest always. Evolution: Avro by name + default + alias, Protobuf by field number + reserved, JSON by required + additionalProperties.
• Backward vs forward vs full. Backward = new reader reads old data → upgrade consumers first. Forward = old reader reads new data → upgrade producers first. Full = both hold → any order. Add _TRANSITIVE to check against all prior versions (not just N−1) whenever old data is still in retention. FULL_TRANSITIVE is the correct default for topics consumed by teams you do not coordinate with.
• Avro evolution template. Add a field → give it a default (backward-compatible add). Rename a field → add aliases: ["old_name"] (never edit the name bare). Remove a field → safe if the reader tolerates its absence. Test evolution by calling schemaless_reader(buf, writer_schema, reader_schema) and asserting resolution succeeds; if it raises, the change is not backward compatible.
• Confluent wire framing. Every Avro/Protobuf/JSON-Schema message on Confluent Kafka is magic_byte(0x00) + schema_id(4-byte big-endian uint32) + payload. Decode with struct.unpack(">I", frame[1:5]) to get the schema ID, fetch (and cache) the writer schema, then deserialize frame[5:]. A non-zero byte 0 means a mis-produced or cross-format message — check it first when a consumer throws a deserialization error.
• Protobuf wire format. Each field = (field_number << 3) | wire_type tag then value. Wire types: 0 varint (int/bool/enum), 1 64-bit, 2 length-delimited (string/bytes/message/packed), 5 32-bit. Recover field number with tag >> 3, wire type with tag & 7. Varints are base-128 little-endian with the high bit as continuation. Reserve field numbers 1–15 (1-byte tag) for the hottest fields.
• Protobuf evolution rules. Rename freely (names never touch the wire). Add with a new, never-used number (old readers skip and preserve it). Remove with reserved <number>; and reserved "<name>"; in the same commit — reusing a number silently misreads old bytes. Use sint32/sint64 for values that go negative (zig-zag), and mark a field optional in proto3 to distinguish "sent zero" from "unset."
• JSON Schema evolution rules. Add fields as optional (not in required) to stay backward compatible; adding to required is the canonical break. Keep additionalProperties: true for forward compatibility; flipping it to false makes any new field a forward break. Renames are breaks with no alias escape hatch — the key is the identity and it is in the payload. Always validate on decode; JSON parses without a schema, so nothing enforces the contract unless you call validate().
• Byte-budget reflex. For a {id, name, active} record: JSON ≈ 36 bytes, Avro ≈ 6, Protobuf ≈ 9 (plus 5-byte Confluent framing for Avro/Protobuf/JSON-Schema serdes). At a billion events/day JSON's repeated field names are a 3–8× tax versus binary. Encode a representative record, count bytes, multiply by event rate, and put the number next to the readability benefit before choosing.
• Presence and defaults. Avro: every field should carry a default so future readers can back-fill it. proto3: unset scalars read as 0/""/false and are indistinguishable from "sent zero" unless marked optional. JSON Schema: absence is legal only if the field is not in required. Model nullable-where-zero-is-meaningful fields explicitly in every format.
• Registry subject strategy. TopicNameStrategy (default) scopes compatibility to <topic>-value; RecordNameStrategy / TopicRecordNameStrategy let multiple record types share one topic with per-record compatibility. Pick the strategy that matches your topic's payload variety — one evolving type per subject keeps the compatibility guarantee honest.
• CI compatibility gate. POST /compatibility/subjects/<subject>/versions/latest with the candidate schema returns {"is_compatible": true|false} for any of the three formats. Wire it into the pull-request pipeline and fail the build on false. This one gate turns "we hope the deploy is safe" into a check, and it is format-agnostic.
• Format migration. Never transcode a topic in place — one partition cannot mix two wire formats safely. Dual-write to a second topic, migrate consumers one group at a time, verify the old topic has no live readers (frozen consumer-group offsets), then stop the old write and delete after retention. A four-phase dual-write, not a flag flip.

Frequently asked questions

Avro vs Protobuf — which should I use for Kafka?

Pick Avro when wire size dominates and you already run the Kafka/Confluent ecosystem: it produces the smallest fully-populated records, its writer/reader schema resolution with defaults and aliases is the most ergonomic evolution model, and Confluent Schema Registry is Avro-native. Pick Protobuf when you want strong typing, gRPC reuse across polyglot services, and number-based evolution: the field number is a permanent contract, renames are free, and optional gives you real presence tracking in proto3. The honest one-liner for avro vs protobuf is that Avro optimises the data-pipeline axis (compact rows, registry-driven evolution, data-lake landing) while Protobuf optimises the service-contract axis (typed messages that also serve RPC). Both are binary, both integrate with the schema registry, and both are excellent — the decision is which axis your workload is bound by. If you have no strong constraint and you are on Confluent Kafka, Avro is the lower-friction default; if your events are shared with gRPC services, Protobuf reuses the same .proto.

What is the difference between backward and forward compatibility?

backward compatibility means a new reader can read data written by an old writer — you upgrade consumers first, and it is the safe default because you usually control consumer rollout. forward compatibility means an old reader can read data written by a new writer — you upgrade producers first, useful when a producer must ship a new field before consumers are ready. The trick to never confusing them is the phrase "new reads old = backward, old reads new = forward," always paired with the deploy order. FULL compatibility means both directions hold, so you can deploy in any order — the most restrictive and the safest. Each mode has a _TRANSITIVE variant that checks the candidate schema against all previous versions rather than only the immediately preceding one, which matters whenever old-version messages are still in retention or tiered storage.

Do I always need a schema registry?

For Avro on Kafka, effectively yes — Avro data carries no field names or types on the wire, so a consumer cannot decode a byte without the exact writer schema, and the registry is how that schema is referenced (via the 4-byte ID in the Confluent framing) and cached. For Protobuf, a registry is strongly recommended but not strictly required to decode, because the compiled .proto descriptor ships with the consumer and unknown fields are skipped-and-preserved; the registry's value is centralised compatibility enforcement, not decode-ability. For JSON Schema, you can decode with no registry at all (JSON is self-describing), so the registry buys you validation and compatibility CI rather than decode-ability. The deeper point is that the registry is the contract enforcement point — it makes "version N+1 is compatible with N" a provable, CI-checkable property rather than a hope, and that value applies to all three formats even where decode-ability does not require it.

How big is JSON compared to Avro and Protobuf on the wire?

For a small record like {"id": 42, "name": "Ada", "active": true}, JSON is about 36 bytes, Avro about 6, and Protobuf about 9 (before the 5-byte Confluent framing that Avro/Protobuf/JSON-Schema serdes all add). The gap is structural, not incidental: JSON repeats every field name as UTF-8 text in every message and stores numbers as decimal strings, while Avro writes values positionally with no names and Protobuf writes a one-byte numeric tag per field. At high volume the difference compounds — a billion events per day at a 3–8× size multiplier is a large, ongoing network-and-storage bill. That wire tax is the main reason high-volume telemetry pipelines move off JSON, and the main reason low-volume, debuggability-first streams happily stay on it. Always encode a representative record, count the bytes, and multiply by your event rate before deciding.

Can Protobuf and Avro handle renaming a field?

Protobuf renames are free — field names never touch the wire, only field numbers do, so changing name to full_name while keeping field number 2 is a source-only edit that leaves every existing byte decoding identically. Avro renames require an alias — the field name is part of resolution, so you rename by adding aliases: ["old_name"] to the new field, which tells the resolution algorithm to map the old writer-schema name onto the new reader-schema field. A bare Avro rename (editing the name with no alias) is a break: it drops the old field and adds a new unmatched one, which the registry's compatibility check rejects. JSON Schema renames are breaks with no escape hatch — the key is the identity and it lives in the payload, so there is no alias or reserved mechanism; a rename is effectively a remove-plus-add and must be handled as a migration.

When should I pick JSON Schema over a binary format?

Reach for JSON Schema when human-debuggability and zero-decode-dependency outweigh wire size. Concretely: low-volume domain events where an engineer routinely reads raw messages in a debugger or kafka-console-consumer; integration boundaries with external partners who cannot run your serde or redeploy on your schedule; audit and compliance streams where the message must be legible to a human reviewer; and configuration or control-plane topics where clarity beats compactness. You keep real contract enforcement — Confluent stores JSON Schema in the same registry with the same backward/forward/full compatibility modes (interpreted through required and additionalProperties) — so the choice is not "structure vs no structure," it is "readable-and-larger vs compact-and-opaque." Avoid JSON Schema for high-volume telemetry, where its repeated-field-name tax is a real cost, and remember that validation is opt-in: JSON parses without a schema, so you must wire validate() into the deserializer or the contract is unenforced.

Practice on PipeCode

• Drill the streaming practice library → for the Kafka serialization, schema-registry, and compatibility-mode scenarios senior interviewers open with when avro vs protobuf comes up.
• Rehearse on the JSON practice library → for the payload-shape, self-describing-parse, and JSON Schema validation problems that decide whether a consumer survives a schema change.
• Sharpen the pipeline axis with the ETL practice library → for the Avro container-file ingestion, schema-drift, and format-migration patterns that turn raw events into governed tables.
• Layer in design drills for the schema-registry topology, subject-strategy, and typed-event-contract questions that separate a format opinion from a defensible architecture.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the three-axis decision matrix and the backward/forward/full compatibility rules against real graded inputs.

Lock in serialization and schema-evolution muscle memory

Docs explain the formats. PipeCode drills explain the decision — when Avro's registry dependency is worth the compact wire, when Protobuf's field number is the contract, when JSON Schema's readability beats the byte tax, and which compatibility mode keeps an independently-deployed fleet from breaking itself. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice streaming problems →
Practice JSON problems →