DEV Community

Cover image for encoding/gob in Go: Binary Serialization Built for Go-to-Go Services
Gabriel Anhaia
Gabriel Anhaia

Posted on

encoding/gob in Go: Binary Serialization Built for Go-to-Go Services

#go #serialization #backend

You have two Go services that talk only to each other. A scheduler
that hands work to a pool of workers. A cache node that ships entries
to a peer. Both ends are Go. Both ends share the same structs, often
from the same internal module. And yet the payload on the wire is
JSON, because JSON is the default everyone reaches for.

There is a format in the standard library built for exactly this
case. encoding/gob is Go's own binary serialization. It was written
for Go programs talking to Go programs, and when that is your whole
world, it does things JSON cannot. It also has one caveat sharp
enough that reaching for it in the wrong place will hurt. This post
is about both.

What gob actually is

gob streams typed values between an Encoder and a Decoder. The
encoder writes a self-describing binary stream: the type definition
travels with the data the first time a type appears, so the decoder
knows how to reconstruct the value without a shared schema file.

package main

import (
    "bytes"
    "encoding/gob"
    "fmt"
)

type Job struct {
    ID       int
    Payload  string
    Priority uint8
}

func main() {
    var buf bytes.Buffer
    enc := gob.NewEncoder(&buf)
    dec := gob.NewDecoder(&buf)

    if err := enc.Encode(Job{ID: 7, Payload: "resize", Priority: 3}); err != nil {
        panic(err)
    }

    var got Job
    if err := dec.Decode(&got); err != nil {
        panic(err)
    }
    fmt.Printf("%+v\n", got)
}
Enter fullscreen mode Exit fullscreen mode

Run it and you get {ID:7 Payload:resize Priority:3}. No struct
tags, no field annotations, no code generation step. You point an
encoder at an io.Writer and a decoder at an io.Reader, and Go
handles the rest.

The stream is stateful. Once a type crosses the wire, its definition
is not sent again on the same stream. So a long-lived connection that
sends a thousand Job values pays the type-description cost once, not
a thousand times. That is a real difference from JSON, where every
message re-sends every field name as text.

gob vs JSON for internal RPC

JSON earns its place at the edges of your system. Browsers speak it,
other-language services speak it, humans read it in a log. None of
that matters between two Go processes you own on both ends.

Here is where gob pulls ahead for the internal case:

  • No field names on the wire. JSON writes "priority" as five bytes of text on every message. gob sends field numbers after the first type description. Repeated messages get smaller.
  • Types the way Go means them. A JSON number decodes to float64 unless you fight it, which quietly breaks large int64 values. gob preserves int64, uint8, time.Time, and the rest with no loss and no tags.
  • No reflection tax you can see. gob caches the encoding plan per type after first use, so a hot loop over the same struct is not re-deriving anything.

Where JSON still wins: anything crossing a language boundary, any
payload a human needs to read, any public API, and anything you want
to curl. If a non-Go client might ever read the bytes, gob is the
wrong tool. That is not a knock on gob. It is the boundary it was
drawn for.

A rough decision line: gob for the wire between services you deploy
together and update together. JSON for the wire between you and
anyone else.

Registering interface types

Structs with concrete fields work with zero setup. The moment a field
is an interface, gob needs help, because the decoder has to know which
concrete type to build behind the interface.

Say your Job carries a Payload that can be one of several shapes:

type Payload interface {
    Kind() string
}

type ResizeImage struct {
    Width, Height int
}

func (ResizeImage) Kind() string { return "resize" }

type SendEmail struct {
    To      string
    Subject string
}

func (SendEmail) Kind() string { return "email" }

type Job struct {
    ID      int
    Payload Payload
}
Enter fullscreen mode Exit fullscreen mode

Encode a Job whose Payload is a ResizeImage and gob will fail
at decode time with a message about the concrete type not being
registered. The fix is gob.Register, called once at startup for
every concrete type that can travel through an interface field:

func init() {
    gob.Register(ResizeImage{})
    gob.Register(SendEmail{})
}
Enter fullscreen mode Exit fullscreen mode

Now the encoder writes the concrete type's name into the stream, and
the decoder looks it up in the registry to build the right value.

func main() {
    var buf bytes.Buffer
    enc := gob.NewEncoder(&buf)
    dec := gob.NewDecoder(&buf)

    out := Job{ID: 1, Payload: ResizeImage{Width: 800, Height: 600}}
    if err := enc.Encode(out); err != nil {
        panic(err)
    }

    var in Job
    if err := dec.Decode(&in); err != nil {
        panic(err)
    }
    fmt.Println(in.Payload.Kind()) // resize
}
Enter fullscreen mode Exit fullscreen mode

Two rules keep this honest. Register the concrete type, not a pointer,
unless you actually send pointers, in which case register the pointer
form too. And put every gob.Register call in the same package on
both ends, so the encoder and decoder agree on the registered set.
The registry is process-global, so registering the same type twice is
fine, but registering two different types under the same name will
panic.

Versioning fields across deploys

Internal services drift. One side ships a new field on Monday, the
other side rolls out Wednesday. gob is built to tolerate that gap, and
the rules are worth knowing before you rely on them.

gob matches fields by name, not by position. That gives you a
forgiving contract:

  • A field the decoder does not have is skipped. The encoder can send a struct with an extra field to an old decoder and the old decoder ignores it.
  • A field the encoder did not send is left at its zero value on the decoder. A new decoder reading old data gets the zero value for the new field.
  • Renaming a field breaks the match. The old name is skipped, the new name arrives at its zero value. A rename is a remove plus an add, so treat it that way.

Picture the encoder side one deploy ahead:

// v2 encoder
type Job struct {
    ID       int
    Payload  string
    TenantID string // new in v2
}
Enter fullscreen mode Exit fullscreen mode 
// v1 decoder, not yet deployed
type Job struct {
    ID      int
    Payload string
}
Enter fullscreen mode Exit fullscreen mode

The v1 decoder reads the v2 stream fine. TenantID is not in its
struct, so gob drops it on the floor. No error. When v1 rolls forward
to v2, TenantID starts arriving and populating. Additive changes are
safe in both directions, which is what you want for a rolling deploy.

The trap: type changes are not additive. If Priority was uint8 in
v1 and you widen it to int in v2, gob will refuse a value that does
not fit or will error on the type mismatch. Changing a field's type is
a breaking change on the wire. Add a new field and migrate rather than
mutating an existing one, exactly as you would with a database column
you cannot afford to break.

If you need fields that never go out, keep them exported. gob ignores
unexported fields entirely. An unexported field is not an error, it
just never crosses the wire and stays at its zero value on the far
side, which surprises people who expected it to travel.

The caveat that decides everything: gob is not cross-language

This is the line that sends people back to JSON, and it should. gob
has no meaningful implementations outside Go. The wire format is
documented, but there is no maintained gob library for Python, Java,
Rust, or JavaScript that you would want in production.

So the boundary is strict. If the byte stream will ever be read by
something that is not a Go program built from compatible types, gob
is the wrong format. That covers public APIs, browser clients,
polyglot service meshes, message queues that other teams consume, and
anything you archive to disk and might read years later from a
different stack.

Inside a fleet of Go services you own end to end, none of that
applies, and gob's downsides evaporate. The version-tolerance rules
above assume both ends share the same Go types anyway, so the format's
Go-only nature is a description of its home, not a limitation you fight.

One more practical note: gob is a Go-runtime format, not a stable
long-term storage format. The team maintains wire compatibility across
Go versions, but treating a gob blob as a decade-long archive format is
asking more of it than it promises. For durable storage that outlives
your current binaries, pick a format with a schema you control.

When to reach for it

Reach for gob when both ends are Go, you deploy them together, and the
payload is internal plumbing rather than a public contract. A worker
pool, a cache-replication link, a snapshot passed between replicas of
the same service, an RPC layer where you own the client and the
server. In those places gob gives you typed, compact, tag-free
serialization out of the standard library with no dependencies to
vendor.

Reach for JSON, protobuf, or anything with a cross-language story the
instant a non-Go reader enters the picture. The test is one question:
will anything that is not your Go code ever read these bytes. If yes,
gob is out. If no, gob is often the least-effort format that also
happens to be the fastest to wire up.

If this was useful

gob sits right on the seam between Go's runtime and Go's type system,
which is why it rewards knowing how the language represents values
underneath. The Complete Guide to Go Programming goes deep on that
layer: reflection, interface internals, and how the encoding packages
build their plans, so the behavior above stops feeling like magic.
Hexagonal Architecture in Go is the companion for keeping a choice
like gob where it belongs: behind an adapter at the edge of your
service, so swapping the wire format later never leaks into your core.

Thinking in Go — the 2-book series on Go programming and hexagonal architecture

Top comments (0)