Avro schemas for Kafka developers: the 10-minute practical guide

#kafka #avro #dataengineering #tutorial

If you work with Kafka long enough, someone puts Avro in front of you. The docs are dense, so here's the working knowledge you actually need: what a schema looks like, the two rules that cause 90% of production errors, and how to debug a binary payload.

Why Avro instead of JSON?

Two reasons: size (binary encoding, field names aren't repeated in every message — a JSON message of 500 bytes often becomes ~100) and contracts (the schema is enforced at produce time, so a malformed message never enters the topic). With a schema registry, consumers always know how to decode what producers wrote — including messages written under older schema versions.

The schema, minimally

{
  "type": "record",
  "name": "OrderCreated",
  "namespace": "com.shop.events",
  "fields": [
    { "name": "orderId", "type": "string" },
    { "name": "amount", "type": "long" },
    { "name": "currency", "type": "string", "default": "KRW" },
    { "name": "couponCode", "type": ["null", "string"], "default": null }
  ]
}

The parts that trip people up:

There is no "optional" keyword. An optional field is a union with null — ["null", "string"] — and the default must be null, with "null" listed first in the union. Order matters; ["string", "null"] with a null default is invalid.
Defaults are not runtime fallbacks. A default is used when a reader decodes data written by an older schema that lacked the field. Producers still must set every field.
Timestamps are long with a logical type: { "type": "long", "logicalType": "timestamp-millis" }.

The two evolution rules that matter

Schema evolution is why Avro exists, and it boils down to:

Adding a field? It must have a default. Otherwise old messages can't be read with the new schema (BACKWARD compatibility breaks — the registry will reject it).
Removing a field? Only remove fields that had a default. Never rename — a rename is a remove plus an add, and old data loses the value silently. Add an alias instead.

Everything else (changing types, reordering unions) — check against your registry's compatibility mode before assuming.

Debugging: "what's actually in this message?"

Binary Avro is unreadable in kafka-console-consumer, and the classic gotcha is the magic byte: messages produced through Confluent serializers carry a 5-byte header (0x00 + 4-byte schema ID) before the Avro payload. If your decoder chokes immediately, that header is usually why.

For quick inspection without spinning up kafka-avro-console-consumer, I use these browser tools (client-side only, nothing uploaded):

Avro decoder — paste base64/hex bytes + the schema, get JSON back
Avro schema generator — paste a sample JSON message, get a starting-point schema with proper null unions
Avro message generator — generate test payloads from a schema

TL;DR

Optional field = ["null", "type"], null first, default null.
New fields need defaults; never rename, alias instead.
Defaults serve schema evolution, not producer laziness.
Decoder failing on byte 0? Strip the 5-byte Confluent header.

DEV Community