- Book: Hexagonal Architecture in Go
- Also by me: Pocket Guides for Developers — the 8-book series
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Your storage port looks clean: Save(ctx, key string, data []byte) error. The domain never touches the filesystem, the S3 adapter sits behind the interface, and the tests use a map. Then someone uploads a 2 GB video. To call Save, the HTTP handler has to produce a []byte, so it runs io.ReadAll(r.Body).
Read what io.ReadAll does: it builds "slices of exponentially growing size, then copy into a perfectly-sized slice at the end." For a moment, the chunks and the final slice are both alive. That 2 GB upload costs you roughly twice its size in heap before your domain code runs a single line. Under a memory limit of a few gigabytes, two uploads at the same time can be enough to get the container OOM-killed.
Blame the signature. Change it and the memory problem goes away for every adapter at once.
The port: io.Reader in, io.ReadCloser out
The fix is to let bytes flow through the port instead of sitting in it. Put takes an io.Reader and reads it to EOF. Get hands back an io.ReadCloser the caller streams and closes.
package storage
import (
"context"
"errors"
"io"
)
var (
ErrNotFound = errors.New("storage: not found")
ErrInvalidKey = errors.New("storage: invalid key")
)
// Store keeps blobs by key. Put reads r to EOF.
// A Put that returns an error must leave no object.
type Store interface {
Put(ctx context.Context, key string,
r io.Reader) (int64, error)
Get(ctx context.Context,
key string) (io.ReadCloser, error)
Delete(ctx context.Context, key string) error
}
The doc comment on Store is part of the port. "A failed Put leaves no object" is a promise every adapter has to keep, and later a test will hold each one to it.
The handler gets shorter too. http.MaxBytesReader caps the body and still returns a reader, so the cap costs no memory:
func (h Handler) upload(w http.ResponseWriter,
r *http.Request) {
body := http.MaxBytesReader(w, r.Body, 5<<30)
key := "uploads/" + r.PathValue("id")
_, err := h.store.Put(r.Context(), key, body)
if err != nil {
http.Error(w, "upload failed",
http.StatusBadRequest)
return
}
w.WriteHeader(http.StatusCreated)
}
In a real service the handler calls a use case and the use case calls Store. The signature stays the same all the way down: the use case passes the reader through and never buffers it.
The snippets below share one package so they stay short. In a real layout the port sits in storage, each adapter gets its own package (storage/disk, storage/s3store, storage/memory), and the contract test lives in storage/storagetest as an exported RunContract. Only the S3 adapter's package imports the AWS SDK, so the domain never does.
A []byte port fills the bucket first. An io.Reader port is a pipe.
Disk adapter: write a temp file, then rename
The local adapter has one job beyond copying bytes: a reader must never open a half-written file. os.CreateTemp in the same directory as the target, then os.Rename onto the final name. Same directory means same filesystem, so on Unix the rename swaps the name in one step. The Go docs are blunt about the exception: "on non-Unix platforms Rename is not an atomic operation."
First, turning a key into a path:
type Disk struct{ root string }
func NewDisk(root string) *Disk { return &Disk{root: root} }
func (d *Disk) path(key string) (string, error) {
k := filepath.FromSlash(key)
if !filepath.IsLocal(k) || filepath.Clean(k) != k ||
k == "." {
return "", ErrInvalidKey
}
return filepath.Join(d.root, k), nil
}
The key check uses filepath.IsLocal (Go 1.20+), which rejects absolute paths and anything that climbs out of the root. The extra Clean and "." checks reject keys like uploads/.. that stay inside the root but resolve to a directory. The docs warn it "is a purely lexical operation" that ignores symlinks, so don't let untrusted users create symlinks under the root.
Then Put, which writes to a temp file and only renames it once everything succeeded:
func (d *Disk) Put(ctx context.Context, key string,
r io.Reader) (int64, error) {
dst, err := d.path(key)
if err != nil {
return 0, err
}
dir := filepath.Dir(dst)
if err := os.MkdirAll(dir, 0o755); err != nil {
return 0, err
}
tmp, err := os.CreateTemp(dir, ".put-*")
if err != nil {
return 0, err
}
done := false
defer func() {
if !done {
os.Remove(tmp.Name())
}
}()
n, err := io.Copy(tmp, ctxReader{ctx, r})
if err == nil {
err = tmp.Chmod(0o644)
}
if err == nil {
err = tmp.Sync()
}
if cerr := tmp.Close(); err == nil {
err = cerr
}
if err != nil {
return n, err
}
if err := os.Rename(tmp.Name(), dst); err != nil {
return n, err
}
done = true
return n, nil
}
CreateTemp makes the file "with mode 0o600 (before umask)", per the os docs, so without the Chmod your web server may not be able to read what you stored. Sync flushes the data before the name points at it. Close returns an error of its own, so the code checks it instead of dropping it. The deferred cleanup removes the temp file on every failure path.
If the process dies mid-upload, though, a .put-* file stays behind under a name no Get will ask for, so sweep old ones at startup. And if the rename itself has to survive a power cut, also sync the parent directory.
Get and Delete are short. Return the *os.File only on success, so you never hand back a typed nil inside an interface:
func (d *Disk) Get(ctx context.Context,
key string) (io.ReadCloser, error) {
p, err := d.path(key)
if err != nil {
return nil, err
}
f, err := os.Open(p)
if errors.Is(err, fs.ErrNotExist) {
return nil, ErrNotFound
}
if err != nil {
return nil, err
}
return f, nil
}
func (d *Disk) Delete(ctx context.Context, key string) error {
p, err := d.path(key)
if err != nil {
return err
}
err = os.Remove(p)
if errors.Is(err, fs.ErrNotExist) {
return nil
}
return err
}
The last piece is the reader wrapper Put copies through:
type ctxReader struct {
ctx context.Context
r io.Reader
}
func (c ctxReader) Read(p []byte) (int, error) {
if err := c.ctx.Err(); err != nil {
return 0, err
}
return c.r.Read(p)
}
ctxReader stops a copy when the request is cancelled. It also hides any WriterTo on the source, so the copy ends up on io.Copy's fixed 32 * 1024 byte buffer. That buffer is the only per-upload memory this adapter needs.
S3 adapter: transfermanager, not the old upload manager
If your S3 code still imports feature/s3/manager, it's on a deprecated module. The aws-sdk-go-v2 release of 2026-02-05 reads: "Deprecate feature/s3/manager, superceded by feature/s3/transfermanager." The maintainers' announcement puts the successor's production release on January 30, 2026. New code should use transfermanager:
type S3 struct {
client *s3.Client
tm *transfermanager.Client
bucket string
}
func NewS3(c *s3.Client, bucket string) *S3 {
tm := transfermanager.New(c,
func(o *transfermanager.Options) {
o.FailTimeout = 30 * time.Second
})
return &S3{client: c, tm: tm, bucket: bucket}
}
Put hands the reader to the uploader, wrapped so it can report how many bytes went through:
func (s *S3) Put(ctx context.Context, key string,
r io.Reader) (int64, error) {
cr := &countingReader{r: r}
_, err := s.tm.UploadObject(ctx,
&transfermanager.UploadObjectInput{
Bucket: aws.String(s.bucket),
Key: aws.String(key),
Body: cr,
})
return cr.n, err
}
type countingReader struct {
r io.Reader
n int64
}
func (c *countingReader) Read(p []byte) (int, error) {
n, err := c.r.Read(p)
c.n += int64(n)
return n, err
}
What does "streaming" mean for an HTTP body you can't seek? Reading the UploadObject source, it reads up to the multipart threshold into memory first. If the body ends before that, it sends one PutObject. Otherwise it pulls parts from a pool of Concurrency+1 buffers of PartSizeBytes each. The defaults in api_client.go are a 16 MiB threshold, 8 MiB parts, concurrency 5 and 10,000 max parts. So memory per upload is bounded by those knobs, not by the file size.
Two consequences of those numbers. First, a non-seekable stream at 8 MiB parts tops out at 10,000 parts, about 78 GiB, because the uploader can only grow the part size when it knows the length up front. Raise PartSizeBytes if you accept bigger streams. Second, the failure promise holds. A single PutObject is all or nothing: "Amazon S3 never adds partial objects," per the PutObject reference. On the multipart path, a read or part error makes the uploader call AbortMultipartUpload before returning, and an unfinished multipart upload is never a readable object.
By default the uploader runs that abort on the context you passed in, which here is the request context. The server cancels that context when the client's connection closes, so a client that hangs up can take the abort down with it. The FailTimeout option gives the cleanup a fresh context so it can "succeed even when the original context is canceled," which is why NewS3 sets it. A process that crashes mid-upload can't abort anything at all, and S3 keeps billing for the parts until you complete or stop the upload. Add the AbortIncompleteMultipartUpload lifecycle rule that page recommends as the backstop for both cases.
Get and Delete call the plain S3 client:
func (s *S3) Get(ctx context.Context,
key string) (io.ReadCloser, error) {
out, err := s.client.GetObject(ctx,
&s3.GetObjectInput{
Bucket: aws.String(s.bucket),
Key: aws.String(key),
})
var nsk *types.NoSuchKey
if errors.As(err, &nsk) {
return nil, ErrNotFound
}
if err != nil {
return nil, err
}
return out.Body, nil
}
func (s *S3) Delete(ctx context.Context, key string) error {
_, err := s.client.DeleteObject(ctx,
&s3.DeleteObjectInput{
Bucket: aws.String(s.bucket),
Key: aws.String(key),
})
return err
}
Imports are aws, feature/s3/transfermanager, service/s3 and service/s3/types from github.com/aws/aws-sdk-go-v2, plus time from the standard library. The errors.As pattern for modeled S3 errors is the one in the SDK's error-handling guide.
One trap on Get: per the GetObject reference, a missing key returns 404 only if the caller has s3:ListBucket. Without it you get 403, and your adapter reports a permissions error where you expected ErrNotFound.
Disk, S3 and the in-memory fake all plug into the same Store interface.
The fake, and one contract test all three must pass
Use-case tests need a store that is fast and has no setup. The fake buffers, which is fine for test-sized data:
type Memory struct {
mu sync.Mutex
objs map[string][]byte
}
func NewMemory() *Memory {
return &Memory{objs: map[string][]byte{}}
}
func (m *Memory) Put(ctx context.Context, key string,
r io.Reader) (int64, error) {
b, err := io.ReadAll(ctxReader{ctx, r})
if err != nil {
return int64(len(b)), err
}
m.mu.Lock()
defer m.mu.Unlock()
m.objs[key] = b
return int64(len(b)), nil
}
Get hands back a new reader over the stored bytes on every call, so two readers never share a position:
func (m *Memory) Get(ctx context.Context,
key string) (io.ReadCloser, error) {
m.mu.Lock()
defer m.mu.Unlock()
b, ok := m.objs[key]
if !ok {
return nil, ErrNotFound
}
return io.NopCloser(bytes.NewReader(b)), nil
}
func (m *Memory) Delete(ctx context.Context, key string) error {
m.mu.Lock()
defer m.mu.Unlock()
delete(m.objs, key)
return nil
}
A fake is only useful if it behaves like the real thing. Write the port's promises down once as a test suite, then run it against every adapter. iotest.ErrReader simulates a client that disconnects halfway:
func RunContract(t *testing.T, newStore func(*testing.T) Store) {
ctx := context.Background()
t.Run("put then get", func(t *testing.T) {
s := newStore(t)
n, err := s.Put(ctx, "a/b.txt",
strings.NewReader("hello"))
if err != nil || n != 5 {
t.Fatalf("put: n=%d err=%v", n, err)
}
if got := read(t, s, "a/b.txt"); got != "hello" {
t.Fatalf("got %q", got)
}
})
t.Run("missing key", func(t *testing.T) {
_, err := newStore(t).Get(ctx, "never-written")
if !errors.Is(err, ErrNotFound) {
t.Fatalf("want ErrNotFound, got %v", err)
}
})
t.Run("failed put leaves nothing", func(t *testing.T) {
s := newStore(t)
body := io.MultiReader(
strings.NewReader("half an upl"),
iotest.ErrReader(errors.New("client gone")))
if _, err := s.Put(ctx, "failed", body); err == nil {
t.Fatal("want error from broken body")
}
_, err := s.Get(ctx, "failed")
if !errors.Is(err, ErrNotFound) {
t.Fatalf("partial object visible: %v", err)
}
})
t.Run("overwrite then delete", func(t *testing.T) {
s := newStore(t)
s.Put(ctx, "k", strings.NewReader("v1"))
s.Put(ctx, "k", strings.NewReader("v2"))
if got := read(t, s, "k"); got != "v2" {
t.Fatalf("got %q, want v2", got)
}
if err := s.Delete(ctx, "k"); err != nil {
t.Fatal(err)
}
_, err := s.Get(ctx, "k")
if !errors.Is(err, ErrNotFound) {
t.Fatalf("after delete: %v", err)
}
})
}
The read helper reads a whole object back and fails the test on any error:
func read(t *testing.T, s Store, key string) string {
t.Helper()
rc, err := s.Get(context.Background(), key)
if err != nil {
t.Fatalf("get %s: %v", key, err)
}
defer rc.Close()
b, err := io.ReadAll(rc)
if err != nil {
t.Fatal(err)
}
return string(b)
}
Wiring the memory and disk adapters in takes a few lines each:
func TestMemory(t *testing.T) {
RunContract(t, func(*testing.T) Store {
return NewMemory()
})
}
func TestDisk(t *testing.T) {
RunContract(t, func(t *testing.T) Store {
return NewDisk(t.TempDir())
})
}
The S3 run skips unless you point it at a bucket, so in CI you can aim it at a test bucket, or at an S3-compatible server you've checked handles multipart uploads:
func TestS3(t *testing.T) {
bucket := os.Getenv("STORAGE_TEST_BUCKET")
if bucket == "" {
t.Skip("STORAGE_TEST_BUCKET not set")
}
cfg, err := config.LoadDefaultConfig(
context.Background())
if err != nil {
t.Fatal(err)
}
c := s3.NewFromConfig(cfg, func(o *s3.Options) {
if ep := os.Getenv("STORAGE_TEST_ENDPOINT"); ep != "" {
o.BaseEndpoint = aws.String(ep)
o.UsePathStyle = true
}
})
RunContract(t, func(t *testing.T) Store {
return NewS3(c, bucket)
})
}
config is github.com/aws/aws-sdk-go-v2/config. The suite belongs next to the port, not next to any adapter: it is the executable version of the doc comment on Store. A new adapter for GCS or Azure Blob is done when it passes this file.
Measure it yourself: the []byte port against the streaming one
Two benchmarks, same disk adapter, 256 MiB per operation. Both read from an endless source of zeros, cut to size:
const size = 256 << 20
type zeros struct{}
func (zeros) Read(p []byte) (int, error) {
clear(p)
return len(p), nil
}
func body() io.Reader { return io.LimitReader(zeros{}, size) }
The first benchmark copies the old port's habit: buffer everything, then hand over the bytes. The second passes the reader straight through.
func BenchmarkBytesPort(b *testing.B) {
s := NewDisk(b.TempDir())
b.ReportAllocs()
for b.Loop() {
data, err := io.ReadAll(body())
if err != nil {
b.Fatal(err)
}
_, err = s.Put(context.Background(), "blob",
bytes.NewReader(data))
if err != nil {
b.Fatal(err)
}
}
}
func BenchmarkStreamPort(b *testing.B) {
s := NewDisk(b.TempDir())
b.ReportAllocs()
for b.Loop() {
_, err := s.Put(context.Background(), "blob",
body())
if err != nil {
b.Fatal(err)
}
}
}
Run go test -bench Port -benchmem. b.Loop needs Go 1.24 or newer. I'm not printing numbers here because I haven't run this on your hardware and Go version. What the code predicts: BenchmarkBytesPort allocates more than 256 MiB per op, because io.ReadAll grows its chunks and then copies them into a final slice. BenchmarkStreamPort allocates kilobytes, dominated by io.Copy's 32 KiB buffer. B/op counts total allocation, not peak resident memory, so pair it with a heap profile if you want the peak.
If this was useful
This post is one port and three adapters. Hexagonal Architecture in Go is the whole picture: how to draw the line between domain and infrastructure in a Go project, where ports and adapters live in the layout, and how persistence plugs in without the domain importing an SDK. If this post made you look twice at a []byte in one of your interfaces, that book is where I wrote down the rest of the reasoning.
Top comments (0)