For Weeks I Logged a CLI as Flaky. The Bug Was `| tail -20`.

Originally published on hexisteme notes.

I had weeks of notes saying a vendor CLI "works sometimes." Availability seemed to depend on
the hour. I'd started routing around it.

The CLI was fine every time. The bug was in how I was watching it.

The idiom

Every one of those observations came through some variant of this:

perl -e 'alarm 45; exec @ARGV' agy models 2>&1 | tail -20

Timeout on the left, truncate on the right. This idiom hung indefinitely three separate times,
in three separate sessions, on two different subcommands.

• A (2026-08-14) — agy models | tail -20, PIDs 96135→96138. No response.
• B (2026-08-14) — a different session. agy --print ... | tail -120, PIDs 87045→87048. Sat there for 81 minutes.
• C (2026-08-15) — agy models 2>&1 | head -20, PIDs 14195→14198. Hung past 33 minutes.

Every one of them, re-run without a pipe, returned in seconds:

agy models > /tmp/agy-check.out 2>&1   # EXIT=0, a few seconds, every time

Finding it

Take the blocked head/tail and look at what it's reading from:

lsof -p 14198 | grep -i pipe
# fd 0 → 0x4881848dc2e3034f

Now find anyone else holding that same pipe address:

ps -ef | grep -i npm            # what else spawned around then?
lsof -p 14251 | grep -i pipe
# fd 5, fd 6 → 0x4881848dc2e3034f      ← same pipe, write end

PID 14251 was npm exec xcodemcp@latest. It had nothing to do with my command. It had spawned
in the same second as the head.

The other two matched the same way. In A, the culprit was npm exec xcodemcp@latest (PID
96158, spawned at 10:40:22 — within seconds of the tail). In B it was
npm exec @modelcontextprotocol/server-redis (PID 87098), confirmed by matching the pipe
address 0xe3e977fcc256b0c7 on both ends.

The mechanism

Three facts, and it falls out:

1. tail -N without -f can't emit anything until EOF. It has to know which lines are the
   last N. head -N looks safe by comparison — it should exit as soon as it's seen N lines —
   but if total output is fewer than N lines, it also waits for EOF. Both are vulnerable.
   (Before case C, my notes only mentioned tail, so the head variant wasn't covered by the
   rule I'd written. The rule was narrower than the mechanism.)

2. EOF arrives when the pipe's write end reference count hits zero — not when your command
   exits. A process spawned around the same moment can inherit that write-end file descriptor
   (a missing close-on-exec, as best I can tell) and hold it open indefinitely.

3. So head/tail blocks forever, waiting for a signal that a completely unrelated process is
   holding hostage.

The kill switch doesn't save you either. alarm 45 reaches the process exec replaced —
the vendor CLI. But the CLI already exited. What's blocking is a descriptor held by a
different process, so raising the timeout does nothing. I'd been raising it for weeks.

One mechanism, two symptoms

This is the part that made it hard to see.

If your caller just waits, you observe an infinite hang. If your caller has its own
timeout, you observe empty output — and empty output reads as "that leg is dead / the
vendor is down." Same bug, two entirely different verdicts, and the second one is the one that
got written into my notes as evidence about a third party.

Case B was worse still. When I killed the process holding the write end, tail finished within
two seconds — and what came out wasn't a successful response. It was
Error: timeout waiting for response, 36 bytes.

The original command had failed, immediately, and said so. That 36-byte failure message then
spent 81 minutes stuck in a pipe.

That's worth stating precisely, because it's not the usual failure mode. The bug did not turn
a failure into a success. It blocked a failure signal from arriving at all. A wrong answer
and no answer are different problems, and the second one is the one that trains you to distrust
a tool that was telling you the truth on time.

The fixes

Don't attach a truncator to a live pipe. Land the output in a file, then cut it.

# Vulnerable — head and tail alike (head waits for EOF when output < N lines)
perl -e 'alarm 45; exec @ARGV' agy models 2>&1 | tail -20
perl -e 'alarm 45; exec @ARGV' agy models 2>&1 | head -20

# Safe — receive completely (process exit closes the fds), then cut
perl -e 'alarm 45; exec @ARGV' agy models > /tmp/agy-check.out 2>&1
tail -20 /tmp/agy-check.out

Structurally better: skip the shell pipe. A runtime that defaults descriptors to
close-on-exec closes the inheritance path entirely. Python has since PEP 446:

import subprocess
r = subprocess.run(["agy", "models"], capture_output=True, text=True, timeout=45)
print(r.stdout[-2000:])

Why "I can't reproduce it" proved nothing

This only fires when some other process — an MCP server, an editor helper, a daemon — happens
to spawn in the same instant. Most of the time nothing does, and the command returns fine.

Which means "not reproducible" was never evidence of absence here. It was evidence that the
race is a race. Weeks of intermittent success is exactly the signature, and I read it as a
property of the vendor instead of a property of my own plumbing.

The portable version:

When a tool feels like it "works sometimes," suspect the observation pipeline that produced
that judgement before you suspect the tool.

Availability may not be an attribute of the tool at all. It may be a state of your machine at
that moment — and if your measurement apparatus is what's flaky, every measurement it produces
is about the apparatus.

Diagnostic recipe

# 1) read-end pipe address of the blocked head/tail
lsof -p <blocked_pid> | grep -i pipe

# 2) find another process holding the same address as its write end
ps -ef | grep -i npm            # check spawn times of candidates
lsof -p <candidate_pid> | grep -i pipe
# matching pipe address on both = confirmed

What would change my mind

The close-on-exec diagnosis is an inference, supported by three independent cases showing
the same correlation — simultaneous spawn plus matching lsof pipe address — not a root cause
I traced to a line of source.

The falsifier is clean: if the same hang reproduces with no pipe at all
(cmd > file 2>&1), this diagnosis is wrong and I need to start over. Across all three cases,
file redirection returned in seconds, every time. That's where the attempts to break it stand
so far.

More notes at hexisteme.github.io/notes.