My Check Passed Ten Times Locally, Failed Ten Over HTTP

Read the exit codes of a shell pipeline:

PIPESTATUS=(23 0)

curl on the left, grep -q on the right. So: the download failed, the search
succeeded. grep found exactly what it was looking for — the page was there, the content
had rendered, everything was fine. And yet the if block around it declared "verification
failed" and turned the publish step red.

The pipeline sank precisely because it did its job. This article is about how that
sentence is possible, and why it is sitting in your verification code as a time bomb.

The incident first

6 August 2026, the editorial-series publishing workflow of this blog. After an article
goes out, one step verifies that the live page is really being served. Back then it
looked like this:

set -euo pipefail
...
for attempt in $(seq 1 30); do
  if curl -fsS --max-time 8 "${url}?editorial=${GITHUB_RUN_ID}-${attempt}" \
    | grep -q '</main>'; then
    ok=1
    break
  fi
  sleep 4
done
if [ "$ok" != 1 ]; then
  echo "::error::Live content verification failed: $url"
  exit 1
fi

Thirty attempts, four seconds apart. The logic is plain: fetch the page, look for
</main>, and if it is there the publish is live. The pages were live. The step still
ended in exit 1.

The fix landed in commit 5e6a89db (PR #78) and turned it into this:

curl -fsS --max-time 8 -o /tmp/editorial-verify.html "$url" \
  && grep -q '</main>' /tmp/editorial-verify.html

The pipe is gone, the body goes to a file, the search is its own command. It has worked
ever since. Next to the commit I also left a note: the failure showed up "when the page
exceeds the 64 KB pipe buffer".

Re-creating the same failure in a lab today, half of that note turned out to be wrong.
The mechanism was right; the culprit was not. Let us build the mechanism first.

The unwritten contract of a pipeline

Between two processes in a pipeline there is an unwritten agreement: the consumer is
under no obligation to wait for the producer to finish. It may walk away whenever it
likes. When it does, the kernel tells the producer with a signal.

pipe(7) puts it plainly: if all file descriptors referring to the read end of a pipe
have been closed, a write triggers a SIGPIPE for the calling process. The same page also
gives the alternative — if "the calling process is ignoring this signal", then write(2)
fails with the error EPIPE.

Two paths, two different endings. signal(7) ties SIGPIPE to the "P1990" (POSIX.1-1990) standard and
lists its default Action as Term; the legend on that page defines the action as
"default action is to terminate the process". So if you are not ignoring the signal, you
die. SIGPIPE is number 13 on most architectures.

Bash keeps its own record: bash(1) states that the return value of a simple command is
its exit status, "or 128+n if the command is terminated by signal n". 128+13 = 141. One
line is enough to see it:

$ ( set -o pipefail; cat /dev/zero | head -c 10 >/dev/null ); echo $?
141

Three runs, three times 141. head took ten bytes and closed; cat learned about it on
its next write and died. Nobody made a mistake here — the system worked as designed.

The side that leaves early: grep -q

grep -q exits on the first match. In verification code that is exactly what you want:
do not scan the whole page end to end, find the thing and close.

But "first match" and "first matching byte" are not the same thing. grep works in
lines. To report a match it has to see that line end. To find out how decisive this is,
an 8 MiB body with the search string placed at 10% of it, with varying newline density.
The producer writes in 4 KiB chunks:

no newline at all (single line)  -> producer: wrote all of it (8388608 bytes) | grep: 0
newline every 10000 bytes        -> producer: EPIPE @ 909312 bytes (10.8%)    | grep: 0
newline every 1000 bytes         -> producer: EPIPE @ 905216 bytes (10.8%)    | grep: 0
newline every 80 bytes           -> producer: EPIPE @ 905216 bytes (10.8%)    | grep: 0

That top line was the surprise of the day. In an 8 MiB body with no newline at all,
grep -q saw the string in the first 7 bytes and went nowhere: it read all 8,388,608
bytes, finished in 11 milliseconds and returned 0. The producer never saw a single EPIPE.
In a single-line body the "end of the first line" is the end of the file, so grep has
nowhere to exit early.

The three lines below it agree with each other: when newlines exist, grep leaves as
soon as it completes the line holding the match, and the producer hears about it at 10.8%
of the body. The string was placed at 10%; the gap is roughly one buffer's worth of bytes
sitting in the pipe after the matching line ends.

For anyone dealing with minified HTML this has an inverted consequence: the line
structure of your page determines the behaviour of your verification pipeline. The page
in question is 263,801 bytes and contains 3,216 newlines; </main> sits at byte 192,811,
that is at 73%. Plenty of newlines, plenty of chances to leave early.

Passing locally, failing over the network

Now the critical measurement. The same pipeline, set up twice: once with the body coming
from a local file, once over the network.

Local file, Linux 6.10.14, GNU grep 3.11, ten runs:

$ for i in $(seq 1 10); do cat live.html | grep -q '</main>'; echo -n " ${PIPESTATUS[0]}"; done
 0 0 0 0 0 0 0 0 0 0

cat came out clean ten times out of ten. No error, no signal, no problem.

The same page, this time over the network, curl 8.14.1, with set -o pipefail, ten runs:

$ for i in $(seq 1 10); do
    ( set -o pipefail; curl -fsS --max-time 20 "$U?lnx=$i" | grep -q '</main>' )
    echo -n " $?"
  done
 23 23 23 23 23 23 23 23 23 23

Ten runs, ten times 23. And curl's own message gives away its intent:

curl: (23) Failure writing output to destination, passed 3458 returned 0

PIPESTATUS then hands over that paradoxical pair: (23 0).

One objection is fair up to here: cat on the left, curl on the right. The producer
changed along with the source, and the two have opposite SIGPIPE dispositions. So here is
the control — same curl, same grep, same shell, only the source is file://:

$ for i in $(seq 1 10); do
    ( set -o pipefail; curl -fsS "file:///tmp/live.html" | grep -q '</main>' )
    echo -n " $?"
  done
 0 0 0 0 0 0 0 0 0 0

Producer held constant, result still inverted: ten clean runs from the local source, ten
runs of 23 over the network. That leaves exactly one thing changing — where the bytes came
from. Flawless locally, failing every single time over the network, not intermittently but
ten out of ten.

The fixed version returned 0 ten times out of ten in the same conditions, so the solution
works. But the answer to "why" was somewhere other than where my commit note put it.

The referee: the pipe's 64 KiB

curl does not die, it returns 23, because it ignores SIGPIPE and reports the write
failure with its own error code. cat does not ignore it, and dies with 141. Both are
branches of the same EPIPE/SIGPIPE fork.

So why does the local cat never see an error? Because it wins the race. The number from
pipe(7) comes in here: since Linux 2.6.11 the pipe capacity is 16 pages, which is
65,536 bytes on a system with 4096-byte pages. Opening a fresh pipe and asking gives the
same answer:

F_GETPIPE_SZ = 65536 bytes

If the producer can dump all of its remaining bytes into that buffer before the consumer
leaves, it feels nothing at all: write returns, the process exits 0. The local cat
flushes 258 KiB in microseconds in large blocks and finishes while grep is still getting
ready to report the match. A body arriving over the network, on the other hand, trickles
in 8 KiB pieces at the speed of TCP. By the time grep sees the match at 73%, curl
still has bytes in hand.

To test that explanation, body and speed held constant, with only the pipe capacity
changed via F_SETPIPE_SZ. The producer writes at full speed in every run, ten runs:

Pipe capacity	Producer's outcome
8,192 bytes	EPIPE @ 212,992 — 10/10
16,384 bytes	EPIPE @ 229,376 — 9/10, EPIPE @ 212,992 — 1/10
32,768 bytes	EPIPE @ 262,144 — 10/10
65,536 bytes	no error — 10/10
262,144 bytes	no error — 10/10

The table says one thing: the referee is the buffer size. At the default 64 KiB the
producer wins ten out of ten; shrink the buffer to 8 KiB and the same body at the same
speed loses ten out of ten. No code changed in between — only the room the producer had
to hide in got smaller.

That is why the episode where
a neighbouring container shrank my pipes eightfold
is the direct sequel to this article. That one described how exceeding the quota handed
out new pipes at 8 KiB instead of 64 KiB, with no error returned at all. The first row of
the table above lands exactly on that situation: when your neighbour shrinks your buffer,
a verification pipeline that ran fine locally for years starts producing false negatives.
Where the two articles meet is this: 64 KiB is not a performance setting, it is the margin
of a race.

Correcting my own note

The note on that commit said "when the page exceeds 64 KB". Feeding the body through a
slow producer instead (8 KiB chunks, 2 ms apart) produced this table — three runs per row:

Body	Producer's outcome
65,536 bytes	EPIPE @ 49,152 — 3/3
98,304 bytes	EPIPE @ 73,728 — 3/3
131,072 bytes	EPIPE @ 98,304 — 3/3
196,608 bytes	EPIPE @ 147,456 — 3/3
262,144 bytes	EPIPE @ 196,608 — 3/3
393,216 bytes	EPIPE @ 294,912 — 3/3

A body of exactly 65,536 bytes gets EPIPE too. The range below the threshold is
measured as well — same slow producer, same 73% position, three runs each:

Body	Producer's outcome
8,192 bytes	no error — 3/3
16,384 bytes	no error — 3/3
32,768 bytes	EPIPE @ 24,576 — 3/3
49,152 bytes	EPIPE @ 40,960 — 3/3
65,536 bytes	EPIPE @ 49,152 — 3/3

A 32,768-byte body — half the buffer — blows up three times out of three. So "exceeding
64 KB" is not a threshold.

The two rows at the top teach something for free: the 8 and 16 KiB bodies never fail, but
not because the buffer protects them. The producer simply writes so few bytes that the job
is over before grep gets around to reporting the match. Small pages hide the problem not
because they are safe, but because they finish early.
On top of that, the sweep with the fast producer zigzagged outright: at 323,584 bytes ten
runs clean, at 335,872 ten runs of 141, at 348,160 again 141, but at 360,448 ten clean
runs again, and 141 once more at 393,216. A non-monotonic series of results, which shows
that the sentence "it breaks above size X" cannot be formed at all.

The accurate sentence is this: it is a race with three inputs — how fast the bytes
arrive, how large each write call is, and how much room is left in the buffer. Size on
its own determines none of them. Testing with a local file tunes all three in your
favour; over the network all three turn against you.

Who reports it, and how

The same event looks completely different depending on who the producer is. From the
measurements:

Producer	SIGPIPE	What the pipeline shows
cat, grep	not ignored	141 (128+13), a silent death
curl 8.14.1 (Linux)	ignored	23, "Failure writing output to destination"
curl 8.7.1 (macOS)	ignored	56, identical message text
Python 3.12 (Linux)	ignored	BrokenPipeError + traceback, exit 1
Python 3.13 (macOS)	ignored	same error + a second flush error, exit 120

Version 8.7.1 of curl on macOS returns 56 rather than 23 in the same scenario — ten
times out of ten, with identical message text. Do not pin that on the version alone: the
transport decides. The same binary returns 23 when it speaks plain HTTP over localhost.
If you are branching on the exit code, that matters: a comparison that is correct in one
environment quietly becomes wrong in another.

Python is its own story. The documentation states the situation outright: a small number
of default handlers are installed, and SIGPIPE "is ignored", so that write errors on pipes
and sockets "can be reported as ordinary Python exceptions". In practice that means:

$ ( set -o pipefail; python3 -c 'for i in range(10**7): print(i)' | head -3 >/dev/null )
Traceback (most recent call last):
  File "<string>", line 2, in <module>
BrokenPipeError: ...
$ echo $?
1

That 1 is not portable, though. The same command on macOS with Python 3.13.3 returns
120, and stderr carries an extra line:

BrokenPipeError: [Errno 32] Broken pipe
Exception ignored on flushing sys.stdout:
BrokenPipeError: [Errno 32] Broken pipe

EPIPE lands twice: once during print, and once while the interpreter tries to flush
sys.stdout on its way out. The second one turns the exit code into 120. On Linux
(Python 3.12, both glibc and musl) the second hit does not happen and the code stays 1.
Same code, same pipe, two different answers.

When your script is invoked as yourscript | head, the user gets a traceback. The Python
docs offer a pattern for it: catch BrokenPipeError, redirect remaining output to
os.devnull, exit 1. The warning in the same document matters just as much — do not set
SIGPIPE's disposition to SIG_DFL to escape this error, because then your program "would
exit unexpectedly whenever any socket connection is interrupted while your program is
still writing to it". Applying the recommended pattern cleared stderr entirely and produced exit 1 in both
environments — on macOS it brought 120 down to 1. The doc's sys.exit(1) is the code of
the cure, not of the symptom.

The setting that completes the story: pipefail

Everything described so far is none of your concern without pipefail. With pipefail
off, a pipeline's exit code belongs to the rightmost command alone — that is, to grep's

1. The producer's death is recorded, but nobody reads the record.

bash(1) changes the rule when pipefail is on: the return status of the pipeline becomes
"the value of the last (rightmost) command to exit with a non-zero status, or zero if all
commands exit successfully". grep 0, curl 23 → pipeline 23. if calls that a failure.

Running the same if block over the network under two shells:

bash -e        (the `run:` default) -> VERIFIED
bash -eo pipefail (shell: bash)     -> FALSE NEGATIVE

This is a direct trap for anyone on GitHub Actions. The workflow syntax documentation
gives the two lines separately: when shell is unspecified on Linux and macOS, the command
run internally is bash -e {0}; when you write shell: bash, it becomes
bash --noprofile --norc -eo pipefail {0}. So pipefail is in the second and absent from
the first.

The consequence: add shell: bash in good faith to a step that has been green for years,
touch nothing else, and your verification can start producing false negatives. In my step
pipefail was already on explicitly through set -euo pipefail, so it had been there from
day one, and it only surfaced once the page grew large enough and arrived slowly enough.

A checklist

Going through your own verification code, look at these in order:

1. Is there a consumer in the pipeline that exits early? Patterns like grep -q, grep -m1, head, head -c, sed -n '1p;1q', tail -n +N | head. If so, the producer's death is on the table.
2. Is pipefail on? set -o pipefail, set -euo pipefail, or shell: bash in Actions. If it is, the producer's death becomes your pass/fail decision.
3. Is the producer fed from the network? curl, wget, kubectl logs -f, ssh host cat, docker logs. The network slows the producer down and makes it lose the race. A test against a local file will not show you this.
4. Does the body contain newlines? In minified single-line JSON or HTML, grep -q cannot exit early, so the problem is invisible on its own. Then one day the server switches to pretty-printing and the failure appears.

On the fix side, my own preference in order:

Remove the pipe. This is the sturdiest. Put the body in a file and make the search its
own command:

out=$(mktemp); trap 'rm -f "$out"' EXIT
curl -fsS --max-time 8 -o "$out" "$url" && grep -q '</main>' "$out"

Do not hardcode /tmp/verify.html: two jobs on the same machine will clobber each other's
file, and a predictable name invites a symlink game on a shared runner. mktemp costs one
extra line.

Both commands then return their own correct status, and the race disappears. This is the
form that returned 0 ten times out of ten. You also keep the downloaded body, whose value
you come to appreciate while debugging.

Pick a consumer that does not exit early. Swapping grep -q for grep -c left cat
with 0 in five runs out of five, because grep -c reads the input to the end. The price is
scanning a large body for nothing. Acceptable on small pages, not on an 8 MiB log stream.

Silence the code as a last resort. || true, or reading ${PIPESTATUS[1]}, works —
but do it knowing what it costs: you also silence the producer's genuine errors. In
verification code I dislike this; verification itself should not be the one place that
swallows failures.

And one habit: when you wonder which member of a pipeline returned what, look at
PIPESTATUS. Seeing that (23 0) pair from the first line of this article took the
diagnosis from half an hour down to half a minute. On zsh the array is named pipestatus.

What I am leaving open

The fix works, but writing this exposed a gap: the similar patterns in the repository have
not been swept one by one yet. Files containing pipefail still hold a few lines ending in
| head -1. Most of them work on small outputs, so the producer wins the race and nothing
shows. The difference between "not showing for now" and "correct" is exactly what this
article is about, and that sweep is next on my list.

Something more general stayed with me as well. A pipeline's exit code carries the outcome
of a race between the speeds of two processes, and the buffer size is that race's margin.
That is a kind of uncertainty we accept while measuring performance — but it is
unacceptable in verification code. Verification is charged with answering "is this
correct?"; a check that ties its answer to how fast bytes arrive is not answering the
question, it is drawing lots.

What a check that passed ten times locally and failed ten times over the network taught
me is this: a test going red does not show that the test found something. It only shows who
won the race that time.

Official Sources

• pipe(7) — the SIGPIPE/EPIPE fork, the 16-page capacity and pipe-max-size
• signal(7) — SIGPIPE's standard and its default "Term" action
• bash(1) — the pipefail rule and the 128+n status of a signal-killed command
• Python signal — SIGPIPE being ignored and the recommended BrokenPipeError pattern
• GitHub Actions workflow syntax — bash -e {0} versus bash -eo pipefail {0}