- Book: The Complete Guide to Go Programming
- Also by me: Hexagonal Architecture in Go — the companion book in the Thinking in Go series
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You wrote the loop everyone writes. A bufio.Scanner, a for, a line counter. It ran clean in tests against a
scanner.Scan()
sample file. Then it hit a real log dump in production and stopped
reading a third of the way through, no panic, no obvious error. The
process exited zero. The downstream job processed a truncated file
and nobody noticed until the numbers didn't reconcile a day later.
This is the most common way bufio.Scanner bites Go developers, and
it bites quietly. The scanner has a maximum token size. When a single
line is longer than that limit, Scan() returns false and the loop
ends early — exactly the same signal it uses for a clean end of file.
If you don't check the error afterward, the truncation is invisible.
The 64KB ceiling
bufio.Scanner reads into an internal buffer. When a token (a line,
by default) doesn't fit, the scanner tries to grow that buffer up to a
maximum. That maximum is bufio.MaxScanTokenSize, which is 64 * 1024,
so 65536 bytes. One line longer than 64KB and the scanner gives up on
that token.
Here is the loop with the mistake most people make:
func countLines(r io.Reader) (int, error) {
sc := bufio.NewScanner(r)
n := 0
for sc.Scan() {
n++
}
return n, nil // <-- the bug
}
The function returns nil for the error no matter what happened. If
the scanner stopped because a line was too long, you never find out.
The count is wrong and the caller trusts it.
Always check Scanner.Err()
Scan() returning false means one of two things: the input ended,
or something went wrong. You tell them apart by calling Err() after
the loop. On a clean end of file, Err() returns nil. On the
oversized-line case, it returns bufio.ErrTooLong.
func countLines(r io.Reader) (int, error) {
sc := bufio.NewScanner(r)
n := 0
for sc.Scan() {
n++
}
if err := sc.Err(); err != nil {
return n, err
}
return n, nil
}
Feed that a file with a 200KB line and you get back the error string
bufio.Scanner: token too long. That is the difference between a
silent data-integrity bug and a loud failure you can act on. Checking
Err() after the loop is not optional. Treat it the way you treat
checking the error on a database query.
Raising the ceiling with Buffer()
If your lines can legitimately be big (JSON-lines with fat records, a
log line carrying a full stack trace, a CSV row with an embedded
blob), raise the limit. Scanner.Buffer takes an
initial byte slice and a maximum size, and you have to call it before
the first Scan().
func countBigLines(r io.Reader) (int, error) {
sc := bufio.NewScanner(r)
buf := make([]byte, 0, 64*1024)
sc.Buffer(buf, 10*1024*1024) // up to 10MB per line
n := 0
for sc.Scan() {
n++
}
return n, sc.Err()
}
The first argument is the starting buffer the scanner reuses. The
second is the hard cap — the scanner will grow up to that size and no
further. Pick a number that fits your real data with headroom, not
math.MaxInt. A ceiling is a guardrail against a malformed 4GB
"line" eating all your memory. If you remove the guardrail entirely,
one bad input file can OOM the process.
Note the order. Buffer() must run before Scan(). Call it after the
loop has started and it does nothing useful.
Split functions: lines are just the default
The line behavior isn't hardcoded into the scanner. It comes from a
split function, and bufio.ScanLines is the default. The standard
library ships four:
-
bufio.ScanLines— one line per token, strips the trailing\nand\r\n. The default. -
bufio.ScanWords— splits on whitespace, one word per token. -
bufio.ScanRunes— one UTF-8 rune per token. -
bufio.ScanBytes— one byte per token.
You pick a non-default one with Split, called before the first
Scan():
func countWords(r io.Reader) (int, error) {
sc := bufio.NewScanner(r)
sc.Split(bufio.ScanWords)
n := 0
for sc.Scan() {
n++
}
return n, sc.Err()
}
The token-size limit still applies to whatever the split function
emits. With ScanWords, a single 64KB-plus "word" trips the same
ErrTooLong. That is rare for words, common for lines.
You can also write your own split function. The signature is
func(data []byte, atEOF bool) (advance int, token []byte, err error).
That is how you parse fixed-width records or split on a custom
delimiter. It is more control than most code needs, but it is there
when a line isn't your unit.
When to drop Scanner for Reader
bufio.Scanner is built for the common case: tokens that comfortably
fit in memory, where a size cap is a feature. When lines can be
arbitrarily long and you refuse to hold a whole one in memory, the
scanner is the wrong tool. Reach for bufio.Reader and
ReadString('\n') instead.
func countAnyLength(r io.Reader) (int, error) {
br := bufio.NewReader(r)
n := 0
for {
line, err := br.ReadString('\n')
if err == io.EOF {
if len(line) > 0 { // trailing line, no newline
n++
}
return n, nil
}
if err != nil {
return n, err
}
n++
}
}
ReadString has no 64KB ceiling — it grows until it hits the
delimiter or end of file, returning what it read even on io.EOF.
The trade is that a hostile input with no newline can still balloon
memory, and you handle the final partial line yourself. For truly
unbounded streaming you go one level lower again, to Read into a
fixed slice, and process chunks without ever materializing a full
line.
The rule of thumb
Three decisions, in order. Do your lines fit in 64KB? Use Scanner
as is, and check Err(). Do some lines run bigger but still bounded?
Use Scanner with Buffer() and a sane cap. Can a line be
arbitrarily large or is the input untrusted? Use bufio.Reader or
raw Read and stream it.
The one line that matters in all three: check the error after the
loop. A for sc.Scan() that ignores sc.Err() is a truncation bug
waiting for a big enough file.
Scanner and Reader are a small slice of what the bufio and io
packages give you, and knowing where one ends and the other begins is
the kind of runtime detail The Complete Guide to Go Programming
digs into. If you want the wider picture, keeping this I/O concern at
the edge of your system instead of smeared through the core,
Hexagonal Architecture in Go is the companion for that boundary
work.
Top comments (0)