When Your Pipeline Goes Silent: The stdin Buffering Trap

You know this one. You wire up tail -f /var/log/app.log | grep error | your-webhook, it works in your terminal, then you deploy it and... nothing. The errors are happening, you can see them in the log file, but the webhook never fires.

It's not grep. It's not the webhook. It's stdout buffering.

When stdout isn't a TTY — which it isn't in a pipe — the C library switches from line-buffered to fully buffered (usually 4 KB). So grep finds the match, writes it to its stdout buffer, and sits there waiting for that buffer to fill. Your webhook sees nothing until 4 KB of errors accumulate. On a quiet system, that's forever.

The fix is one flag:

# GNU grep
grep --line-buffered error

# BSD/macOS grep
grep --line-buffered error  # same flag works

# Or the portable way
stdbuf -oL grep error

stdbuf -oL forces line buffering on stdout for any command. The -oL means "stdout, line buffered." There's also -o0 for unbuffered (every write is a syscall) and -eL for stderr.

If you're on a system without stdbuf (some minimal containers), unbuffer from the expect package does the same thing by faking a TTY:

unbuffer tail -f /var/log/app.log | grep error | your-webhook

The rule of thumb: any time you pipe into something that consumes line-by-line (grep, awk, sed, your own script reading stdin), and the producer isn't interactive, check buffering. Add --line-buffered or wrap the consumer in stdbuf -oL.

I've lost hours to this. Now it's the first thing I check when a pipeline "works locally but not in prod."