DEV Community: Mohammad Fikri

Why Go's encoding/csv Burns 540MB on 5M Rows (and How I Fixed It)

Mohammad Fikri — Tue, 18 Aug 2026 15:06:51 +0000

TL;DR: Go's built-in encoding/csv.Read() allocates string slices and strings for every single row, generating 10M+ allocations and burning 540 MB on a 5M-row file. I built go-zerocsv to replace this with in-place typed scanning (Record.Scan) and a compacting 4 KB buffer, keeping memory flat at ~5 KB with zero allocations per record on the hot path.

If you parse or write large CSV files in Go, you've probably run into the standard library's memory appetite.

A few days ago, I was profiling a batch pipeline processing a 5-million-row CSV export. The Go process was consuming hundreds of megabytes of RAM, and the garbage collector was eating a noticeable chunk of CPU time.

When you fire up go tool pprof on a heavy encoding/csv run, you immediately see the culprits dominating the allocation graph: runtime.makeslice and runtime.rawstring. The standard library's API is fundamentally designed around allocating new heap objects for every single field and row.

To solve this, I wrote go-zerocsv — a zero-dependency CSV parser and writer that runs with 0 heap allocations per record on the hot path and keeps memory usage at a constant ~5 KB, whether you're streaming 1,000 rows or 10,000,000 rows.

Here is a breakdown of why encoding/csv allocates so much, how go-zerocsv eliminates those allocations, and the engineering decisions behind it.

Why `encoding/csv` Allocates So Much

Look at the reader's method signature in encoding/csv:

func (r *Reader) Read() (record []string, err error)

Every time Read() is called, two things happen:

It allocates a new []string slice for the row (runtime.makeslice).
It copies each parsed byte slice into a newly allocated heap string (runtime.rawstring).

If you read a 5,000,000-row CSV file with 6 columns:

10,000,000+ allocations occur just from slice headers and string conversions.
~540 MB of cumulative heap allocations are pushed onto the garbage collector.

Memory Model Comparison

encoding/csv (Millions of individual heap objects):
Each row allocates a slice, and each cell allocates a heap string:
Row 1 -> []string{"1", "Alice", "98.5", "true"} (4 heap allocations)
Row 2 -> []string{"2", "Bob", "85.2", "false"} (4 heap allocations)
Result: Over 10,000,000 pointer objects created across the run.
go-zerocsv (One reusable 4 KB buffer):
A single internal buffer holds raw bytes and is compacted between records:
[ 1,Alice,98.5,true\n ] -> In-place Scan(&id, &name, &score, &active)
Result: The buffer is reused for the next row. Total heap usage stays flat at ~5 KB.

The writer has a similar problem. Because Writer.Write([]string) only accepts strings, writing typed values (int, float64, time.Time, bool) forces you to call strconv.Itoa, strconv.FormatFloat, or fmt.Sprintf first — each creating temporary throwaway strings for every cell in every row.

The Benchmark Numbers

Here is a direct comparison running on an AMD Ryzen 5 8400F (linux/amd64, Go 1.26):

Reading a 5,000,000-Row CSV (6 columns per row)

Package	Time	Throughput	Memory (B/op)	Allocations
`encoding/csv` (stdlib)	492 ms	292 MB/s	540 MB	10,000,014
`go-zerocsv`	302 ms	474 MB/s	5.0 KB	12

go-zerocsv parses the entire 5M-row file in ~300ms while holding just 5 KB in RAM.

Writing a 5,000,000-Row CSV (mixed types + RFC3339 timestamp)

Package	Time	Memory (B/op)	Allocations
`encoding/csv` (stdlib)	1,119 ms	272 MB	19,999,917
`go-zerocsv`	708 ms	4.3 KB	6

Before vs. After: Side-by-Side

Reading & Parsing Rows

// ❌ BEFORE (encoding/csv): Allocates strings on Read(), then allocates during parsing
for {
    record, err := r.Read() // Allocates []string + strings
    if err == io.EOF { break }
    id, _ := strconv.Atoi(record[0])
    score, _ := strconv.ParseFloat(record[2], 64)
    active, _ := strconv.ParseBool(record[3])
    process(id, record[1], score, active)
}

// ✅ AFTER (go-zerocsv): 0 heap allocations on the hot path
for {
    rec, err := r.Read()
    if err == io.EOF { break }
    if rec.IsFirst() { continue } // Header helper

    var id int
    var name string
    var score float64
    var active bool

    // Scans in-place directly from reader buffer
    if err := rec.Scan(&id, &name, &score, &active); err != nil {
        log.Fatal(err)
    }
    process(id, name, score, active)
}

Full Code Examples

1. High-Performance Reading with `Record.Scan`

Read() returns a lightweight Record struct. You can scan fields directly into variables in-place, mirroring the familiar database/sql pattern:

package main

import (
    "fmt"
    "io"
    "strings"

    zerocsv "github.com/fikrimohammad/go-zerocsv"
)

func main() {
    data := `id,name,score,active
1,Alice,98.5,true
2,Bob,85.2,false
`
    r := zerocsv.NewReader(strings.NewReader(data))

    for {
        rec, err := r.Read()
        if err == io.EOF {
            break
        }
        if err != nil {
            panic(err)
        }

        if rec.IsFirst() {
            continue // skip header
        }

        var (
            id     int
            name   string
            score  float64
            active bool
        )

        // Parses numbers, booleans, and strings in-place (0 allocs)
        if err := rec.Scan(&id, &name, &score, &active); err != nil {
            panic(err)
        }

        fmt.Printf("id=%d name=%s score=%.1f active=%t\n", id, name, score, active)
    }
}

If you don't need Scan, you can access fields with rec.String(i) or copy field bytes into a caller-owned slice using rec.Bytes(i, dst).

2. Zero-Allocation Writing

zerocsv.Writer writes tagged value-type Column structs instead of []string. Reusing a single slice across loop iterations makes the write loop allocation-free:

package main

import (
    "os"
    "time"

    zerocsv "github.com/fikrimohammad/go-zerocsv"
)

func main() {
    w := zerocsv.NewWriter(os.Stdout)
    defer w.Flush()

    // Reuse this slice across all rows
    row := make([]zerocsv.Column, 5)

    for i := 0; i < 100_000; i++ {
        row[0] = zerocsv.ColumnInt(i)
        row[1] = zerocsv.ColumnString("Alice")
        row[2] = zerocsv.ColumnFloat64(98.5)
        row[3] = zerocsv.ColumnBool(true)
        row[4] = zerocsv.ColumnString(time.Now().Format(time.RFC3339))

        if err := w.Write(row...); err != nil {
            panic(err)
        }
    }
}

3. Custom Types without Reflection

If you have custom domain types (like dates or status enums), you don't need reflection or any interface boxing. Implement FieldScanner (for reading) and FieldValuer (for writing):

type Date time.Time

// ScanCSV parses raw bytes into Date with 0 allocations
func (d *Date) ScanCSV(field []byte) error {
    t, err := time.Parse("2006-01-02", string(field))
    if err != nil {
        return err
    }
    *d = Date(t)
    return nil
}

// AppendCSV formats Date directly into the writer's internal buffer
func (d Date) AppendCSV(dst []byte) ([]byte, error) {
    return time.Time(d).AppendFormat(dst, "2006-01-02"), nil
}

Usage:

// Reading:
var birthday Date
err := rec.Scan(&id, &name, &birthday)

// Writing:
err := w.Write(
    zerocsv.ColumnString("Alice"),
    zerocsv.ColumnValuer(birthday),
)

How It Works Internally

1. Reusable Compacting Buffer

Instead of reading the entire file into memory or slicing new buffers constantly, the reader keeps a single reusable buffer (default 4 KB). As records are parsed, processed bytes are shifted and new chunks are read in. If a single CSV line exceeds 4 KB, the buffer grows to accommodate it, and trims back to 4 KB once that line is done.

2. In-Place Quote Decoding

When a CSV field contains escaped quotes (e.g. "He said ""hello"""), standard parsers allocate a clean string. go-zerocsv decodes escaped quotes in-place inside the raw byte buffer and adjusts slice bounds, avoiding intermediate strings.

3. 48-Byte `Column` Value Struct

The Column struct is designed to fit in 48 bytes on 64-bit architecture:

type Column struct {
    valuer FieldValuer // 16 bytes (interface for custom types)
    s      string      // 16 bytes (string or zero-copy byte view)
    n      uint64      // 8 bytes  (ints, uints, floats, bools)
    kind   ColumnKind  // 1 byte   (+ 7 bytes padding)
}

Numbers, booleans, and floats are stored directly in n using bit manipulation (e.g. math.Float64bits), so creating a ColumnInt or ColumnFloat64 never allocates on the heap.

4. Zero Allocations for Formatted Writing

The writer maintains a small reusable 32-byte scratch buffer. When ColumnInt(12345) is passed, it uses strconv.AppendInt(scratch[:0], 12345, 10) and streams the bytes straight into the bufio.Writer — no intermediate string created.

Strict Conformance with `encoding/csv`

A fast parser is useless if it misparses edge cases. We fuzzed go-zerocsv against Go's encoding/csv over 3,000,000+ iterations using Go native fuzzing:

Handles standard RFC 4180 rules, CRLF vs LF, multi-line quoted fields, and lazy quotes (WithLazyQuotes).
Matches encoding/csv's non-fatal ErrFieldCount behavior.
Validates delimiters to 7-bit ASCII (WithDelimiter) to prevent multi-byte UTF-8 stream corruption.
Includes WithMaxBuffer(n) to guard against denial-of-service from malformed single-line files.

When Is Standard `encoding/csv` Still Fine?

To be fair: if you are writing a quick one-off CLI script, parsing a small 50-row country code lookup table, or loading a mock fixture in a unit test, encoding/csv is completely fine. The few dozen allocations will complete in a fraction of a millisecond and won't noticeably affect your application.

go-zerocsv is built for scenarios where throughput and memory predictability matter:

High-volume ETL and batch ingestion pipelines
Streaming database dumps or generating multi-gigabyte CSV exports
Microservices processing concurrent user CSV uploads under strict memory limits (e.g., small Kubernetes pods)
Real-time event logging where GC pause spikes directly degrade p99 latencies

Summary

If your Go service processes CSVs at scale, go-zerocsv offers an easy drop-in performance win:

~60% faster reads (~475 MB/s).
Flat 5 KB memory regardless of file size.
Zero allocations on the hot path.
No external dependencies (standard library only, Go 1.18+).

The code and benchmarks are available on GitHub:

Repo: github.com/fikrimohammad/go-zerocsv
Docs: pkg.go.dev/github.com/fikrimohammad/go-zerocsv

Feedback, issues, and PRs are welcome!

go-typedpipe: A Typed, Context-Aware Pipe for Go

Mohammad Fikri — Mon, 06 Apr 2026 05:56:50 +0000

Background

Go channels are one of the best things about the language. But the moment you need context cancellation, error propagation, and safe concurrent shutdown all at once, a simple chan T starts asking you to write a lot of code just to use it correctly. A common pattern looks something like this:

out := make(chan Result, len(urls))
errc := make(chan error, 1)

// Producer
go func() {
    defer close(out)
    for _, url := range urls {
        select {
        case <-ctx.Done():
            errc <- ctx.Err()
            return
        default:
        }
        resp, err := fetch(ctx, url)
        if err != nil {
            errc <- err
            return
        }
        select {
        case out <- Result{Data: resp}:
        case <-ctx.Done():
            errc <- ctx.Err()
            return
        }
    }
}()

// Consumer
for {
    select {
    case <-ctx.Done():
        // Do something
    case err := <-errc:
        // Do something with the error
    case result := <-out:
        // Do something with the result
    }
}

This works. But if you're not careful, it's easy to introduce bugs:

Double-close panics — closing a channel twice crashes the program
Goroutine leaks — the writer forgets to close, readers block forever
Lost values — values buffered before close are never read
Zombie producers — the writer keeps running after the consumer has already failed

The core problem is that Go channels are low-level primitives. They're powerful, but they leave all the hard parts entirely to the developer.

The Inspiration: `io.Pipe`

Before reaching for a channel, consider io.Pipe. It has an elegant design that solves many of the problems above.

io.Pipe is purpose-built for streaming bytes between goroutines — reading a file, proxying an HTTP body, piping command output. It handles backpressure, idempotent close, and error propagation cleanly:

pr, pw := io.Pipe()

go func() {
    defer pw.Close()
    io.Copy(pw, file) // stream file bytes to the reader
}()

io.Copy(dst, pr) // consume on the other end

But it only works with []byte. The moment your data is a struct, you're out of luck — io.Pipe can't help you there.

That's exactly the gap go-typedpipe fills — taking the same design philosophy as io.Pipe and making it work for any type T.

Enter go-typedpipe

go-typedpipe is io.Pipe, but for any type T.

w, r := typedpipe.New[Event]()

go func() {
    defer w.Close()
    for _, event := range events {
        if err := w.Write(ctx, event); err != nil {
            return
        }
    }
}()

err := r.ReadAll(ctx, func(event Event) error {
    return process(event)
})

No separate error channel. No manual select on every send. No serialization. The same guarantees as io.Pipe — plus buffering, context-awareness, and a drain guarantee.

What it adds over a raw channel

	`chan T`	`go-typedpipe`
Context-aware blocking	Manual `select` on every send/receive	Built into `Write` and `Read`
Close error propagation	Not supported	`CloseWithError` propagates to all consumers
Safe concurrent close	Panics on double-close	Idempotent, safe to call multiple times
Drain guarantee	Values may be lost after close	All buffered values remain readable after close
Consumer loop	Boilerplate `for range` or `select`	`ReadAll` encapsulates the loop

How It Works

Understanding the internals will make you a better Go developer, not just a better user of this library. Let's walk through the key decisions one by one.

Under the hood, the pipe holds two channels:

type pipe[T any] struct {
    ch   chan T        // carries values from writers to readers
    done chan struct{} // closed once on shutdown
    once sync.Once
    err  pipeError
}

Why `ch` is never closed

ch is never closed. This is the most important design decision in the library. In Go, sending to a closed channel causes an immediate panic — and there's no way to recover from it gracefully. Here's what that looks like:

ch := make(chan int, 1)
close(ch)
ch <- 1 // panic: send on closed channel

In a concurrent setting with multiple writers, this becomes a real danger. Two goroutines racing to close the same channel will also panic:

go func() { close(ch) }() // goroutine 1
go func() { close(ch) }() // goroutine 2 — panic: close of closed channel

go-typedpipe sidesteps the problem entirely by never closing ch. Instead, shutdown is signaled by closing done — a separate zero-value channel that carries no data, only a signal. Writers and readers select on done to know when to stop. ch stays open for the lifetime of the pipe, so there's no window for a send-on-closed panic, no matter how many goroutines are writing concurrently.

You might wonder: is it safe to never close ch? Yes — Go has a garbage collector that automatically cleans up memory that is no longer used by the program. Think of it like this: as long as something in your code is still holding a reference to an object, the GC will leave it alone. Once nothing references it anymore, the GC will reclaim the memory — no manual cleanup needed. This applies to channels too. Once the pipe goes out of scope and no goroutines are holding references to it, the GC will automatically clean up ch along with the rest of the struct. You can read more about how Go's GC works in the official guide. Not closing a channel never causes a memory leak — it only means receivers won't get the zero-value signal that closing normally provides. In this library, done takes over that responsibility entirely.

Why `Write` uses two selects

The double select in Write exists to give shutdown priority. To understand why, you need to know one thing about Go's select: when multiple cases are ready at the same time, Go picks one at random. That means in a single select, even if done is already closed, Go might still choose to send to ch:

// Dangerous — Go might pick p.ch <- v even after Close() is called
select {
case <-p.done:
    return
case p.ch <- v: // could still be picked randomly
    return
}

The fix is a non-blocking pre-check before the main select. If the pipe is already closed, we return immediately without ever touching ch:

// Priority check: bail immediately if already closed.
select {
case <-p.done:
    return p.err.Load()
case <-ctx.Done():
    return ctx.Err()
default: // not closed, continue
}

// Safe to proceed with the blocking send.
select {
case <-p.done:
    return p.err.Load()
case <-ctx.Done():
    return ctx.Err()
case p.ch <- v:
    return nil
}

How the drain guarantee works

The drain guarantee is handled in Read. When done fires, Read does one non-blocking attempt to pull a buffered value before returning the close error. This means values written before Close are never lost — readers will consume everything in the buffer before seeing the close error.

How errors are propagated safely

Errors are stored atomically so that concurrent goroutines can safely read and write the close error without data races. The first error set always wins — subsequent calls to CloseWithError are silently ignored.

Real Examples

Using `ReadAll`

ReadAll is the right choice for the consume-all case. It handles ErrPipeClosed internally — returning nil on normal close — so callers only see real errors. It also closes the pipe automatically when it returns.

type Result struct {
    URL        string
    StatusCode int
    Body       []byte
}

func scrape(ctx context.Context, urls []string, w typedpipe.Writer[Result]) {
    defer w.Close()
    var wg sync.WaitGroup
    for _, url := range urls {
        wg.Add(1)
        go func(url string) {
            defer wg.Done()
            req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
            if err != nil {
                w.CloseWithError(fmt.Errorf("build request %s: %w", url, err))
                return
            }
            resp, err := http.DefaultClient.Do(req)
            if err != nil {
                w.CloseWithError(fmt.Errorf("fetch %s: %w", url, err))
                return
            }
            defer resp.Body.Close()
            body, _ := io.ReadAll(resp.Body)
            if err := w.Write(ctx, Result{
                URL:        url,
                StatusCode: resp.StatusCode,
                Body:       body,
            }); err != nil {
                return
            }
        }(url)
    }
    wg.Wait()
}

func main() {
    ctx := context.Background()
    urls := []string{
        "https://example.com",
        "https://example.org",
        "https://example.net",
    }

    w, r := typedpipe.New[Result](typedpipe.WithBufferSize(len(urls)))
    go scrape(ctx, urls, w)

    err := r.ReadAll(ctx, func(result Result) error {
        if err := saveToDatabase(result); err != nil {
            return fmt.Errorf("save %s: %w", result.URL, err)
        }
        log.Printf("saved %s (%d)", result.URL, result.StatusCode)
        return nil
    })
    if err != nil {
        log.Fatal("scraper stopped:", err)
    }
}

Important: always use defer w.Close() in the writer goroutine. If the writer exits without closing — for example when its context is canceled — readers will block forever. Since Close is idempotent, calling it after CloseWithError is always safe.

If saveToDatabase fails, ReadAll propagates that error back to the writer via CloseWithError — so the scraper stops fetching on its next Write call.

Using `Read`

Use Read when you need branching logic between reads — for example, routing results differently based on status code.

for {
    result, err := r.Read(ctx)
    if err != nil {
        if !errors.Is(err, typedpipe.ErrPipeClosed) {
            log.Fatal("reader stopped:", err)
        }
        break
    }
    switch {
    case result.StatusCode == http.StatusOK:
        if err := saveToDatabase(result); err != nil {
            r.CloseWithError(fmt.Errorf("save %s: %w", result.URL, err))
            return
        }
        log.Printf("saved %s (%d)", result.URL, result.StatusCode)
    case result.StatusCode >= 500:
        log.Printf("server error %s (%d), retrying later", result.URL, result.StatusCode)
        scheduleRetry(result.URL)
    default:
        log.Printf("skipping %s (%d)", result.URL, result.StatusCode)
    }
}

When using Read directly, you're responsible for distinguishing ErrPipeClosed (normal exit) from a real error. ReadAll handles this internally — one reason to prefer it for the straightforward case.

Performance

The design decisions in "How It Works" aren't free — the double select, the done channel check, and the atomic error storage all add overhead over a raw channel. Here's what that costs in practice:

BenchmarkPipe_WriteRead/buffer_64-14      27531162    128.9 ns/op    0 B/op    0 allocs/op
BenchmarkPipe_WriteRead/buffer_1024-14    33059798    109.3 ns/op    0 B/op    0 allocs/op
BenchmarkPipe_ReadAll/buffer_64-14        27784093    130.7 ns/op    0 B/op    0 allocs/op
BenchmarkPipe_ReadAll/buffer_1024-14      33114181    108.6 ns/op    0 B/op    0 allocs/op

Zero heap allocations per operation — the pipe struct, channel, and synchronization primitives are allocated once at construction, not on every Write or Read. The extra nanoseconds come from the shutdown-priority select and context checks. That's the deliberate tradeoff: a small, predictable CPU cost in exchange for correctness guarantees that are very hard to get right with raw channels.

When Not to Use It

go-typedpipe is a synchronization primitive for in-process goroutine communication. It's not the right tool when:

You need to broadcast the same value to multiple consumers — each value is delivered to exactly one reader
You need durable messaging across process restarts
You need cross-process communication — use a message broker instead

Getting Started

go get github.com/fikrimohammad/go-typedpipe/v2

Requires Go 1.18 or later. Full documentation and source at github.com/fikrimohammad/go-typedpipe.

Channels are one of Go's best features — but like any primitive, they come with sharp edges when used in complex concurrent scenarios. go-typedpipe doesn't replace channels; it wraps them with the patterns you'd write anyway, so you can focus on your business logic instead of synchronization correctness. If it saves you from one subtle goroutine leak or one send-on-closed panic, it's done its job.