48-Hour Field Notes: Exit Code 137, a Zero-Byte Output File, and the Pipeline That Swallowed Both

A nightly job of mine reads a day of application logs, asks a model to tag each line as noise or signal, and writes the keepers to a summary file. It ran on a small always-on box, called an HTTP endpoint, and printed done into a log I only open when something feels wrong. On the second morning of a 48-hour experiment, the summary file was zero bytes and the log said done — twice.

So which one is lying: the job, the shell, or me? That question cost me two days and three wrong theories, and the answer was cheaper than any of them: my pipeline was reporting the exit status of tee, not of the work.

The naive version, which I still think looks fine

Here is the shape I started with — a loop over record ids, each one posted to a model endpoint, with output streamed through tee so I could watch progress scroll by.

#!/usr/bin/env bash
# naive-run.sh — the version that lied to me
for id in $(cat ids.txt); do
  curl -sS -X POST "$BASE_URL/chat" \
    -H "Content-Type: application/json" \
    -d "{\"input\": \"$id\"}" \
    | tee -a out.jsonl
done
echo done

Do you see the bug? I didn't, for a full day. tee -a exits 0 as long as it can open its output file, so every iteration looked successful even when curl was killed mid-flight, and echo done ran unconditionally at the end. Add set -euo pipefail at the top and this same script tells a completely different story.

The proof fits in two lines

$ false | tee /dev/null; echo "status=$?"
status=0

$ set -o pipefail; false | tee /dev/null; echo "status=$?"
status=1

That is the entire mystery of the zero-byte file: the pipe was healthy, the writer was fine, and the producer had been killed before it wrote anything worth keeping.

What actually killed the producer

Once pipefail was in place, the reported status was 137. That number is not an application error code; it is 128 + 9, and signal 9 is SIGKILL. Nobody asks politely with SIGKILL — the kernel OOM killer, a cgroup limit, or an impatient supervisor did it. Two checks separated those candidates for me:

• journalctl -k --since "24 hours ago" | grep -i -E "killed process|oom" for kernel-level evidence with a timestamp.
• cat /sys/fs/cgroup/memory.events (cgroup v2) for an oom_kill counter that increments exactly when your process dies.

If that counter moved, you were not slow, throttled, or rate limited — you were over a memory ceiling you never measured. Honest caveat: inside some containers memory.max reads max while the host enforces a limit somewhere else, so the counter is more trustworthy than the limit file.

The preflight probe I now run before row one

Most of my lost hours went into discovering the ceiling at row 300 instead of at second zero. So the runner now refuses to start unless a probe passes, and it prints what it found, because future me will not remember any of it.

#!/usr/bin/env bash
# preflight.sh — abort early, loudly, with reasons
set -euo pipefail

say() { printf '%s %s\n' "[preflight]" "$*"; }

mem_max=$(cat /sys/fs/cgroup/memory.max 2>/dev/null || echo unknown)
mem_cur=$(cat /sys/fs/cgroup/memory.current 2>/dev/null || echo unknown)
say "cgroup memory.max=$mem_max current=$mem_cur"

if [ "$mem_max" != "unknown" ] && [ "$mem_max" != "max" ]; then
  avail=$(( mem_max - mem_cur ))
  if [ "$avail" -lt 100000000 ]; then
    say "REFUSING: less than ~100MB of cgroup headroom"; exit 3
  fi
fi

df -Pk . | awk 'NR==2 { if ($4 < 200000) { print "[preflight] REFUSING: low disk"; exit 3 } }'
command -v curl >/dev/null || { say "REFUSING: no curl on PATH"; exit 3; }
[ -r ids.txt ] || { say "REFUSING: ids.txt missing"; exit 3; }

say "ok: $(wc -l < ids.txt) ids queued"

Exit code 3 is deliberate here: it means "refused to start", which is a completely different signal from "started and failed", and my alerting treats those two cases differently.

The checkpoint that makes being killed survivable

A SIGKILL is not a bug you fix once; it is weather. The real fix is making death cheap — append one JSON line per completed record, flush and fsync it, and skip finished ids on the next run.

import json, os, pathlib

OUT = pathlib.Path("out.jsonl")

def done_ids() -> set[str]:
    if not OUT.exists():
        return set()
    ids = set()
    for line in OUT.read_text(encoding="utf-8").splitlines():
        try:
            ids.add(json.loads(line)["id"])
        except (json.JSONDecodeError, KeyError):
            continue          # a torn final line is expected after SIGKILL
    return ids

def append(record: dict) -> None:
    with OUT.open("a", encoding="utf-8") as f:
        f.write(json.dumps(record, ensure_ascii=False) + "\n")
        f.flush()
        os.fsync(f.fileno())   # survive the next kill

Two details earn their keep. fsync after every record costs throughput but keeps recovery honest, and skipping the malformed final line means one torn write cannot poison an entire resume. Then reconcile at the end: count unique ids in out.jsonl, compare against lines in ids.txt, and exit non-zero when they disagree instead of printing done.

The exit-code table I keep next to the job

Status	Meaning	First thing to check
0	Clean exit	Did pipefail actually apply?
1	Generic failure	stderr, not the log file
2	Shell misuse	quoting, missing argument
126 / 127	Not executable / not found	PATH inside cron, not your interactive shell
137	SIGKILL	cgroup oom_kill, kernel log
141	SIGPIPE	a downstream head closed the pipe
143	SIGTERM	supervisor or deploy restarted you

Where the free tier fit, and where it did not

This is where MonkeyCode genuinely mattered to the experiment, so let me be narrow and concrete. I used its free model access for the classification calls and its free server option to host the runner, because the whole point of a 48-hour experiment was to spend time rather than budget.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What that gave me was a place to run a checkpointed, restartable batch job and an endpoint for high-volume, low-stakes tagging where an occasional retry is harmless. What it did not give me — and what I will not pretend otherwise — is a resource envelope I could plan against in advance. I did not know the memory ceiling on day one, which is precisely why preflight.sh exists in the final version.

Who should not use this shape of workflow: anyone who needs p99 latency guarantees, anyone processing regulated data that cannot leave an approved boundary, and anyone whose job cannot be made idempotent. If a rerun can double-charge a customer or double-send an email, fix that before you optimize for free.

What I would repeat, and what I would change

1. Keep the preflight probe. Ten lines of bash beat six hours of confusion every time.
2. Never trust a log line that ends in done. Trust a reconciliation count instead.
3. Put set -euo pipefail in every script that touches data, money, or cron.
4. Store the exit-code table beside the job, not in a wiki nobody opens during an incident.
5. Change: read the resource ceiling before writing the loop, not after the third kill.

The zero-byte file was not a model problem, a network problem, or a rate-limit problem. It was a shell reporting the health of the pipe's last segment while the first segment was being shot in the head. If you want to run a similar experiment without a card on file, the free model access and free server option are what I used — but bring preflight.sh with you, and tell me what your oom_kill counter says.

Comments

[Launch Gate]

The deepest bug in the naive version isn't the missing pipefail - it's that "done" was a claim, not a measurement. Even with pipefail set, a loop that processes 1900 of 2000 ids and dies on the last curl leaves you with a partial file and a nonzero exit in a log you only open when something feels wrong. The pin that closes the class is a count assertion at the end: ids in, rows out, compare. One line - test "$(wc -l < ids.txt)" -eq "$(wc -l < out.jsonl)" - turns "the pipeline felt fine" into a checked invariant, and it also catches the case pipefail alone misses: curl -sS exits 0 on an HTTP 500 unless you add --fail, so a whole night of 500s would have streamed error bodies into out.jsonl with every status green. The probe-before-row-one habit generalizes too: the preflight I'd add is a single canary id through the entire pipe, asserting one well-formed row lands before the loop starts. Two days of debugging, prevented by asserting on outputs instead of trusting statuses.