When Your Script Dies with EPIPE: The Bug Is Two Functions Upstream

You're staring at a Python script that pipes output to grep or awk or any filter. It works fine in your shell. It fails in CI with a traceback pointing to a write call, something like:

OSError: [Errno 32] Broken pipe

The instinct is to look at line 47 where the write happens. The instinct is wrong.

What Actually Happens

The EPIPE error fires at the writer because the reader closed its end of the pipe first. The reader in your pipeline — grep, head, awk, whatever — reached its input limit and exited. Your script kept writing. The kernel said "no more readers, stop."

The bug isn't at the write. The bug is upstream in two ways:

1. The reader has a finite consumption rate. head -n 100 closes its end after 100 lines. grep exits on first match with -m 1. awk might finish a pattern and close. Your writer doesn't know this is coming.

2. Buffering lies to you. Pipes have a kernel buffer — typically 64KB on Linux. Your writer fills that buffer, then blocks. Meanwhile the reader is already gone. When the buffer drains, the writer wakes up to a closed pipe and gets EPIPE.

The Fix: Handle the Reader's Lifecycle

import subprocess
import signal
import os

# Option 1: Ignore EPIPE and let the reader drive
proc = subprocess.Popen(['awk', 'NR>1{print}', 'data.csv'], 
                        stdout=subprocess.PIPE, stderr=subprocess.PIPE)

# Your script's logic here
for chunk in iter(lambda: sys.stdin.read(8192), ''):
    if proc.poll() is not None:
        break  # reader exited, stop writing
    proc.stdin.write(chunk.encode())

# Option 2: in bash, handle it explicitly
set -o pipefail
grep pattern large-file.txt | awk '{print $2}' || {
    ret=$?
    if [[ $ret -eq 141 ]]; then
        # 141 = 128 + 13 (SIGPIPE)
        # Reader exited before writer finished - this is OK
        :
    fi
}

The set -o pipefail approach isn't enough by itself — you still need to catch the case where the writer gets EPIPE before the pipeline overall exits.

The Debugging Rule

When you get a broken pipe error, look at the downstream command in your pipeline first. Ask:

• Does it have an input limit? (head -n, grep -m, tail -n)
• Does it exit early on some condition?
• What happens if input is shorter than expected?

The traceback points at where the symptom is, not where the cause is.

Practical Check

If you're building pipelines in scripts, add this before you blame the writer:

# Check if the downstream tool exits early
timeout 5 grep -m 1 pattern file.txt && echo "grep exited cleanly"

If it exits before your writer is done, you found your culprit.

---

This comes up often enough that it's worth building the habit: EPIPE is a reader problem, not a writer problem.