- Book: The Complete Guide to Go Programming
- Also by me: Hexagonal Architecture in Go — the companion book in the Thinking in Go series
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You need to gzip a file, hash its contents on the way through,
and upload the result to S3. In a lot of languages that means
three buffers, two temp files, and a helper class that ties them
together. In Go it is one pipeline of wrappers and a single call
to io.Copy. No temp files. No buffer the size of the input.
The reason that works is a design decision the Go team made
early and never walked back: keep the core I/O interfaces down
to a single method. io.Reader has one method. io.Writer has
one method. Everything else in the standard library, and most of
the ecosystem, composes around those two.
The two interfaces the whole ecosystem agrees on
Here is the entire contract. Copied from the io package, not
paraphrased:
type Reader interface {
Read(p []byte) (n int, err error)
}
type Writer interface {
Write(p []byte) (n int, err error)
}
That is it. Read fills a byte slice you hand it and reports
how many bytes it wrote and whether it hit an error. Write
takes a byte slice and reports how many bytes it accepted.
Because the contract is this small, a huge number of types
satisfy it without ever importing io or knowing about each
other. *os.File, *bytes.Buffer, strings.Reader,
net.Conn, *gzip.Writer, *bufio.Reader, *tls.Conn,
http.Request.Body, crypto/sha256's hash. None of them
coordinated. They all agreed on Read or Write and that was
enough.
io.Copy: the function that only knows two methods
Once every source is an io.Reader and every sink is an
io.Writer, moving bytes between any pair of them is one
function:
func Copy(dst Writer, src Reader) (written int64, err error)
io.Copy does not care what dst and src actually are. It
loops: read a chunk from src, write it to dst, repeat until
src returns io.EOF. A fixed-size buffer moves the bytes, so
copying a 4 GB file uses a few kilobytes of memory, not four
gigabytes.
Watch how many unrelated pairs this one signature covers:
// file to network socket
io.Copy(conn, file)
// HTTP response body to stdout
io.Copy(os.Stdout, resp.Body)
// in-memory buffer to a file
io.Copy(file, bytes.NewReader(data))
// network socket to a hash
io.Copy(sha256.New(), conn)
Four combinations, one function, zero special cases. The file
never learns about the socket. The socket never learns about the
hash. They meet at the Read/Write boundary and that is the
only thing they share.
Wrapping: a Reader that takes a Reader
The composition trick is that most wrappers are themselves a
Reader or a Writer that hold another one inside. A gzip
writer is an io.Writer that compresses and forwards to the
io.Writer underneath it. A bufio reader is an io.Reader that
buffers and pulls from the io.Reader underneath it.
So you stack them like pipe fittings. Compress on the way out to
a file:
f, err := os.Create("out.gz")
if err != nil {
return err
}
defer f.Close()
zw := gzip.NewWriter(f) // Writer wrapping the file's Writer
defer zw.Close()
// anything written to zw is compressed, then lands in f
_, err = io.Copy(zw, src)
return err
gzip.NewWriter(f) returns a *gzip.Writer. It is an
io.Writer. It wraps f, which is also an io.Writer. You
write to the outer one, bytes flow compressed into the inner
one. io.Copy at the top has no idea gzip is in the chain. It
sees a Writer and writes to it.
Stacking wrappers into a pipeline
Now the opening example, for real. Read a file, count its bytes,
hash it, and compress it, in a single pass. Each concern is one
wrapper, and they nest:
func archive(srcPath, dstPath string) (string, error) {
src, err := os.Open(srcPath)
if err != nil {
return "", err
}
defer src.Close()
dst, err := os.Create(dstPath)
if err != nil {
return "", err
}
defer dst.Close()
// build the write side: gzip -> file
zw := gzip.NewWriter(dst)
defer zw.Close()
// tee the source through a hasher as we read it
hasher := sha256.New()
tee := io.TeeReader(src, hasher)
if _, err := io.Copy(zw, tee); err != nil {
return "", err
}
if err := zw.Close(); err != nil {
return "", err
}
sum := hex.EncodeToString(hasher.Sum(nil))
return sum, nil
}
io.TeeReader(src, hasher) returns an io.Reader that reads
from src and, as a side effect, writes every byte it reads
into hasher. So one io.Copy reads the file once, feeds the
hash, and writes the compressed output to disk. One pass. Bounded
memory. The hash is exact because it saw every byte that moved.
Swap dst for a net.Conn and you are streaming the compressed,
hashed archive straight over a socket. Nothing else in the
function changes, because a socket is an io.Writer too.
Interfaces built from smaller interfaces
Go does not stop at one method. When two small interfaces come up
together often, the io package composes them into a larger one
by embedding, not by growing the originals:
type ReadWriter interface {
Reader
Writer
}
type ReadCloser interface {
Reader
Closer
}
io.ReadWriter is not a new contract. It is Reader and
Writer glued together. A net.Conn satisfies it because it has
both Read and Write. This is the inverse of what large
frameworks do: instead of one fat interface everyone must
implement, you get tiny interfaces that combine when you need the
combination. A function that only reads asks for io.Reader. A
function that reads and closes asks for io.ReadCloser. It never
asks for more than it uses.
That last part is the actual design rule, and it has a name in Go
circles: accept the smallest interface that does the job. A
function that takes io.Reader instead of *os.File can be
called with a file, a network body, a string, or a test fixture,
with no change.
Why this makes tests trivial
The small-interface habit pays its biggest dividend in testing.
If your function takes an io.Reader, you never need a real file
to test it. strings.NewReader hands you a Reader backed by a
string:
func CountLines(r io.Reader) (int, error) {
sc := bufio.NewScanner(r)
n := 0
for sc.Scan() {
n++
}
return n, sc.Err()
}
func TestCountLines(t *testing.T) {
in := strings.NewReader("a\nb\nc\n")
got, err := CountLines(in)
if err != nil {
t.Fatal(err)
}
if got != 3 {
t.Fatalf("got %d, want 3", got)
}
}
No temp file, no t.TempDir, no cleanup. The same CountLines
works in production against a 2 GB log file streamed off disk,
because *os.File is also an io.Reader. The function has one
input type and it is the narrowest one that fits.
Why the standard library stayed this small
It would have been easy for Go's authors to add methods over the
years. A ReadAll on the interface. A Seek. A Peek. Every
one would have broken the deal, because every method you add to an
interface is a method every implementer must now provide. Widen
io.Reader by one method and you exclude every type that had the
old shape.
So the extra behavior lives in free functions and optional
interfaces instead. io.ReadAll(r) is a function that takes a
Reader, not a method on it. Seeking is a separate io.Seeker
interface you assert for when you need it:
if s, ok := r.(io.Seeker); ok {
s.Seek(0, io.SeekStart)
}
The base contract stays at one method, and the type opts into
more only if it can. This is the whole reason a gzip.Writer
written in 2011 still slots into a pipeline you write today. The
interface it depends on never moved.
Small interfaces are cheap to implement, cheap to satisfy by
accident, and cheap to combine. That is the trade Go made: push
the surface area down to one method, and let composition do the
work that inheritance and fat interfaces do elsewhere. Once you
see Read and Write as the two sockets everything plugs into,
most of the standard library stops looking like a catalog and
starts looking like one pattern repeated.
If this was useful
The io package is the clearest example of a habit that runs
through all of Go: keep the contract small, then compose. The
Complete Guide to Go Programming goes into how these interfaces
work at the runtime level — the type/value pair behind every
interface, why the assertion in the seeker example is cheap, and
how io.Copy picks its buffer. Hexagonal Architecture in Go
takes the same idea up a level, using narrow interfaces as the
ports that keep your domain from leaking into files, sockets, and
frameworks.
Top comments (0)