DEV Community

Neural Download
Neural Download

Posted on

Protobuf: Why Google's Servers Don't Speak JSON

#protobuf #protocolbuffers #grpc #wireformat

https://www.youtube.com/watch?v=OsyKxWxGtiI

Your API returns a user. Three fields — an ID, a name, a bit for active. That exact payload, as compact JSON, is forty-one bytes on the wire. As protobuf, it's twelve. But compactness is the least interesting thing about protobuf. The real reason Google built it is something JSON fundamentally cannot do.

The JSON bill you pay on every request

Take this payload:

{"id":12345,"name":"Alice","active":true}
Enter fullscreen mode Exit fullscreen mode

Twelve bytes of that are actual data. The other twenty-nine are JSON describing itself — quotes, colons, commas, and the key names id, name, and active spelled out in every single message. Every request. Every response. Forever.

You can compress it. You can minify it. It still has to spell itself out on the wire because every reader has to parse a self-describing document.

The real problem isn't bytes — it's fragility

A week later you add a field: email. You ship the new server. But an old client out on someone's phone still only knows three fields. It gets the new payload and hits "email": "a@b.c".

What happens next depends entirely on the library. It might crash. Silently drop the field. Corrupt a nearby field. Reject the whole message. JSON itself has no opinion — there's no contract telling the client how to walk past something unknown.

This is the wall Google hit at scale. And the answer wasn't a smaller JSON. It was a format where the wire itself tells the reader how to skip something it has never seen.

Put the schema on both sides, not in the bytes

A protobuf message is defined by a .proto file:

message User {
  int32  id     = 1;
  string name   = 2;
  bool   active = 3;
}
Enter fullscreen mode Exit fullscreen mode

That schema gets compiled into code on both sides. It is never sent over the wire. So what ends up on the wire?

For id = 12345:

  • One byte tag: "field one, type varint"
  • Two bytes for 12345 as a varint

For name = "Alice":

  • One byte tag: "field two, type length-delimited"
  • One length byte: 5
  • Five bytes: A l i c e

For active = true:

  • One byte tag: "field three, type varint"
  • One byte: 1

Twelve bytes. Same data. JSON wrote forty-one bytes to describe its own structure. Protobuf wrote zero.

The varint — small numbers, small bytes

That "two bytes for 12345" is a varint. Protobuf slices an integer into groups of seven bits. Each group goes into a byte. The top bit of the byte is a continuation flag — one means "more bytes coming," zero means "this is the last one." A reader walks one byte at a time and stops when the flag clears.

Small numbers use one byte. Huge numbers use more. You never pay for bits you don't need.

The tag byte is doing two jobs

That "one byte tag" is where the magic lives. The bottom three bits encode the wire type. The remaining bits encode the field number.

tag = (field_number << 3) | wire_type
Enter fullscreen mode Exit fullscreen mode

Wire types are:

  • 0 — varint (int32, int64, bool, enum)
  • 1 — fixed 64-bit (double, fixed64)
  • 2 — length-delimited (string, bytes, sub-message)
  • 5 — fixed 32-bit (float, fixed32)

This split is what makes protobuf evolvable.

Unknown field? The wire type tells you how to skip

Back to the schema evolution scenario. Old client, new message with an extra email field. Watch the parser:

  • Tag → field 1, varint. Known. Parse ID.
  • Tag → field 2, length-delimited. Known. Parse name.
  • Tag → field 3, varint. Known. Parse active.
  • Tag → field 4. Never heard of it. But wire type says length-delimited — so read the length, skip that many bytes, continue.

No crash. No guess. No corruption. The unknown field just passes through. Some runtimes will even preserve the raw bytes of unknown fields so the client can re-serialize the message and send it back untouched.

You can add fields. You can rename fields — the name was never on the wire. You can remove a field and the reader just keeps going. The one rule: don't reuse a field number for a different meaning, which is why every protobuf schema pins a number next to every field.

What's actually on the line under your gRPC calls

Every gRPC call on the planet uses this format. It's what moves messages inside Kubernetes service meshes. It's on the wire between Google's own data centers. It's how Android push notifications get to your phone.

Not because twelve bytes is smaller than forty-one. Because the bytes were designed so tomorrow's schema can't break yesterday's code.

Top comments (0)