- Book: The Complete Guide to Go Programming
- Also by me: Hexagonal Architecture in Go — the companion book in the Thinking in Go series
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You've seen the memory graph. A Go service that uploads files to
S3 sits flat at 40MB of heap, then someone uploads a 2GB video and
the pod gets OOM-killed. The trace is easy to read after the fact:
io.ReadAll on the request body, then a bytes.NewReader handed to
the S3 client. The whole payload lived in RAM at once, twice over.
Go's io package has three small helpers that make that whole class
of bug go away. None of them are new. None of them are clever. They
just let bytes flow through your program instead of piling up in it.
io.Pipe connects a writer to a reader without a buffer. io.TeeReader
splits a stream so you can hash it while it's already moving.
io.LimitReader caps how much you'll accept before you even start.
Here's how they fit together.
io.LimitReader: cap the read before it hurts you
Start with the defensive one. Any time you read from something you
don't control (an HTTP body, a socket, a file someone else wrote),
you're trusting the other side not to send more than you can hold.
io.LimitReader removes that trust.
func decodeBody(r io.Reader) (Payload, error) {
limited := io.LimitReader(r, 1<<20) // 1 MiB
var p Payload
if err := json.NewDecoder(limited).Decode(&p); err != nil {
return Payload{}, err
}
return p, nil
}
io.LimitReader(r, n) returns a reader that hands back at most n
bytes, then reports io.EOF as if the stream ended. The JSON
decoder never sees byte 1,048,577. A client that streams a
never-ending body can't drive your heap to the moon.
There's a catch worth knowing. A LimitReader that hits its cap
returns io.EOF, which to the decoder is indistinguishable from a
body that legitimately ended early. If you need to tell "the client
sent exactly 1 MiB of valid JSON" apart from "the client sent 5 GiB
and we cut it off," give it one extra byte and check:
func decodeBody(r io.Reader, max int64) (Payload, error) {
limited := io.LimitReader(r, max+1)
buf, err := io.ReadAll(limited)
if err != nil {
return Payload{}, err
}
if int64(len(buf)) > max {
return Payload{}, errors.New("body too large")
}
var p Payload
return p, json.Unmarshal(buf, &p)
}
For HTTP handlers specifically, the stdlib gives you
http.MaxBytesReader, which does the same capping but returns a
typed error and signals the client with a 413. Reach for that in
handlers. io.LimitReader is the general-purpose version for every
other reader you touch.
io.TeeReader: read the stream once, use it twice
Now the tee. You're uploading a file and you also need its SHA-256,
so the other side can verify it. The naive version reads the bytes
for the hash, then reads them again for the upload. Two passes, and
if the source is a network stream, you can't rewind it anyway.
io.TeeReader(r, w) wraps a reader so that every byte read from it
is also written to w. The reader is the primary path; the writer
is a side channel that sees the same bytes go by.
func uploadWithHash(
dst io.Writer, src io.Reader,
) (string, error) {
h := sha256.New()
tee := io.TeeReader(src, h)
if _, err := io.Copy(dst, tee); err != nil {
return "", err
}
return hex.EncodeToString(h.Sum(nil)), nil
}
io.Copy pulls bytes from tee and pushes them to dst. Every
byte it pulls also lands in the hash. When the copy finishes, the
hash has seen the whole stream, and you never held more than
io.Copy's internal 32KB buffer at any moment. One pass, no rewind,
constant memory.
The order matters. TeeReader writes to w as a side effect of
being read. If nobody reads the tee, the hash stays empty. The
io.Copy is what drives it. A TeeReader you construct and never
read is a no-op.
You can stack these. Cap the read and hash it in the same chain:
tee := io.TeeReader(io.LimitReader(src, 1<<20), h)
The limit sits closest to the source, so the hash only ever sees
bytes you agreed to accept.
io.Pipe: connect a writer to a reader with no buffer
The last one is the odd shape. Sometimes you have an API that wants
to write (a json.Encoder, a csv.Writer, tar, gzip), and
another API that wants to read (an HTTP request body, an uploader).
There's no shared buffer to hand between them. io.Pipe builds the
bridge.
io.Pipe() returns a connected *PipeReader and *PipeWriter.
Bytes written to the writer come out of the reader. There's no
internal buffer at all: a Write blocks until a Read consumes it.
That backpressure is the point. The producer can't outrun the
consumer, so memory stays flat no matter how big the payload gets.
The classic use is streaming a generated body into an HTTP request
without building the whole thing first:
func streamUpload(
ctx context.Context, url string, rows []Row,
) error {
pr, pw := io.Pipe()
go func() {
enc := json.NewEncoder(pw)
for _, row := range rows {
if err := enc.Encode(row); err != nil {
pw.CloseWithError(err)
return
}
}
pw.Close()
}()
req, err := http.NewRequestWithContext(
ctx, http.MethodPost, url, pr,
)
if err != nil {
return err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
return nil
}
The goroutine writes JSON rows into pw. The HTTP client reads them
out of pr as it sends the request over the wire. A row is encoded,
shipped, and dropped. The next row starts. You could stream a
million rows through a few kilobytes of live memory.
Two rules keep io.Pipe honest.
The write side runs in its own goroutine. It has to. Write blocks
until the reader consumes, and the reader here is the HTTP client
running on the main path. If you wrote to pw on the same goroutine
that reads pr, both sides would wait on each other and deadlock.
Close with the error, not just close. If the producer fails halfway
through, call pw.CloseWithError(err). That makes the reader's next
Read return that exact error instead of a clean io.EOF, so the
consumer knows the stream is broken rather than finished. A plain
pw.Close() on a failed encode would tell the HTTP client "body
done, all good," and you'd ship a truncated payload as if it were
complete.
Putting the three together
The shapes compose. Say you're proxying an upload: read a bounded
amount from the client, hash it for an integrity check, and stream
it onward to object storage, all in one pass.
func proxyUpload(
ctx context.Context, dst io.Writer, body io.Reader,
) (string, error) {
h := sha256.New()
capped := io.LimitReader(body, 100<<20) // 100 MiB
tee := io.TeeReader(capped, h)
if _, err := io.Copy(dst, tee); err != nil {
return "", err
}
sum := hex.EncodeToString(h.Sum(nil))
return sum, nil
}
LimitReader guards the ceiling. TeeReader feeds the hash. io.Copy
moves the bytes to dst. Peak memory is one 32KB copy buffer plus
the hash state, whether the upload is 1KB or 100MB. There's no
ReadAll anywhere, so there's no payload sitting in the heap waiting
to get you OOM-killed.
The mental model that ties it together: in Go, a stream is a thing
you pull bytes through. You don't hold it. io.Reader and
io.Writer are the two ends. LimitReader, TeeReader, and Pipe
are the fittings that let you branch, cap, and connect those ends
without ever materializing the whole payload. Once you see request
bodies and file uploads as flows instead of blobs, the OOM class of
bug stops being something you can write.
What to check in your codebase
Three greps, same energy as any Go audit.
-
io.ReadAllon anything you don't control the size of. Each one is a memory bound set by the sender, not by you. Wrap the source inio.LimitReaderorhttp.MaxBytesReaderfirst. - Any place you read a stream twice to hash-and-send. That's a
io.TeeReaderwaiting to happen, and it removes a full pass over the data. - Code that builds a big
bytes.Bufferonly to hand it to a reader. If a writer-shaped API feeds a reader-shaped API,io.Pipeconnects them directly and holds the buffer flat.
None of these three helpers are more than a few lines in the stdlib.
That's the point. The composition is where the value lives, and
Go's io interfaces were designed so the small pieces snap together.
If this was useful
Streaming I/O is one of those corners of Go that looks trivial until
a 2GB upload takes down a pod, and then it's suddenly the most
interesting thing in the codebase. The Complete Guide to Go
Programming works through io.Reader/io.Writer from the ground
up — the interface contracts, how io.Copy and bufio behave under
the hood, and why the composition patterns above hold constant
memory. Hexagonal Architecture in Go shows where to draw these
streaming boundaries so your handlers, services, and storage adapters
each own the right slice of the pipe.
Top comments (0)