48-Hour Field Notes: The Helper Worked in My Terminal. /bin/sh Was dash.

Have you ever watched a helper script succeed in an interactive terminal and then fail the moment another process invoked it? I spent forty-eight hours inside that loop, and the failure looked like a cursed remote box instead of a shell dialect. The job itself was tiny: scan a few rotating logs, print the latest error cluster, and exit non-zero when that cluster grew. I did not need another dashboard for that work, only a script that survived both my laptop and a shared remote environment.

What I was actually trying to ship

I wanted a log-triage helper that a Python watchdog could call every minute without me babysitting stdout. The helper had to run on my laptop and on a free remote server when I wanted the same checks away from my desk. I used MonkeyCode's free model access and free server option to draft the first version while already logged into that box.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The generated draft looked confident, used arrays, and leaned on bracket tests that I type without thinking. I pasted it, ran it, and watched it print a clean cluster from a sample nginx error log. Why would I distrust a script that had just worked in front of me?

Hours 0–8: trust the interactive shell

I started the way most of us start, by rerunning the same file in the same terminal until the output felt boring. bash triage.sh /var/log/app/error.log kept returning the cluster I expected, including the non-zero exit when I appended a noisy burst. I added set -x, stared at expanding arrays, and decided the logic was sound enough to wire into a one-minute Python caller.

The Python side was a short subprocess wrapper with shell=True, because the generated snippet included a pipeline. Locally that wrapper still passed, which only made the later failure on the remote box more insulting. On the remote box the same wrapper printed : not found, then a syntax error on a parenthesis, then a silent zero exit. Have you noticed how a zero exit from a broken script is worse than a noisy crash?

I blamed PATH first, because that cheap story always feels productive when you are tired and remote. I printed which python3, which bash, and echo $PATH on both machines and found nothing exotic enough to justify the errors. I blamed log rotation next, then newline translation, then a missing grep -P on the server. None of those guesses survived a careful second look at the actual stderr from the wrapper.

Hours 8–24: blame the box, not the grammar

The free remote environment made a convenient villain when the same file behaved like two different programs. Maybe /tmp was tiny, maybe file watchers were capped, or maybe the log path was a stale symlink. I ran df -h /tmp, ulimit -n, and ls -li on the log file, and those numbers were ordinary. I even recreated the sample log with printf so I could stop accusing rotation and start accusing the runtime.

I then compared process trees while the Python wrapper ran, because argv usually tells on you before metrics do. ps -o pid,ppid,args -p $PID showed /bin/sh -c ./triage.sh on the server, which should have been the whole story. On my laptop a login-shell habit leaked bash into tests I did not measure, so the clue stayed blurry. Why would /bin/sh matter at all when the file I kept rerunning was already named triage.sh?

I also wasted time on Python buffering, because a previous week had trained me to expect unbuffered stdout drama. python3 -u changed nothing, and stdbuf -oL changed nothing, because the script was not hanging in a pipe. The script was parsing itself with the wrong grammar, which is a quieter failure than a blocked file descriptor.

The command that ended the argument

The next morning I ran three commands I should have run before writing a single function:

ls -l /bin/sh
readlink -f /bin/sh
head -n 1 triage.sh

On the remote box /bin/sh pointed at dash, and the script itself had no shebang at the top. My interactive tests had been bash triage.sh, which never consulted /bin/sh even once. Python's subprocess with shell=True had consulted it every time, so the broken server story collapsed. POSIX sh was doing exactly what POSIX sh is specified to do with bash dialect.

Here is a reduced fragment of the draft that dash rejected and bash accepted without complaint:

# example fragment — this is bash, not POSIX sh
set -euo pipefail
patterns=( "upstream timed out" "connect() failed" "no live upstreams" )
logfile="${1:?logfile required}"

if [[ ! -r "$logfile" ]]; then
  echo "cannot read $logfile" >&2
  exit 2
fi

cluster=$(grep -E "$(IFS='|'; echo "${patterns[*]}")" "$logfile" | tail -n 20)
mapfile -t lines <<< "$cluster"

pipefail, [[ ]], arrays, mapfile, and <<< are all bash dialect, not portable sh. Dash answered with syntax errors or with a [[ command that did not exist on the PATH. Once I saw that split clearly, those forty-eight hours felt both obvious in hindsight and painfully preventable. Do you still trust a generated script just because it ran once under your own prompt?

A reproducible artifact

I wanted a check I could rerun on any box, including one I did not own and did not configure. The artifact below is a POSIX-friendly rewrite plus a detector that fails loudly when the runtime shell disagrees. If the detector and the helper disagree, I treat that as a failed test rather than a mysterious host personality.

Detect the runtime before you trust a snippet

# save as detect-shell.sh and run with: sh detect-shell.sh
# example only — verify on your box before you wire it into cron
set -eu

echo "0=$0"
echo "SHELL=${SHELL:-unset}"
echo "pid=$$"

ps -o pid,ppid,comm,args -p $$ 2>/dev/null || true
ls -l /bin/sh
readlink -f /bin/sh 2>/dev/null || true

# Cheap dialect probe: dash and busybox sh will not treat [[ as grammar.
if [ -n "${BASH_VERSION:-}" ]; then
  echo "dialect=bash BASH_VERSION=$BASH_VERSION"
else
  echo "dialect=not-bash (treat generated bashisms as hostile)"
fi

Run it three ways, because that is where the lie hides from interactive demos:

bash detect-shell.sh
sh detect-shell.sh
python3 -c 'import subprocess; subprocess.check_call(["sh", "detect-shell.sh"])'

If those three prints disagree, your helper is not portable yet, no matter how pretty the first draft looked. I now treat disagreement as a failing test, not as a mysterious server personality that only hates my account. Have you ever shipped a filename that implied bash while the caller quietly started dash?

POSIX rewrite of the log cluster helper

#!/bin/sh
# triage.sh — POSIX sh, example helper for a rotating error log
# usage: triage.sh /path/to/error.log
set -eu

logfile="${1:-}"
if [ -z "$logfile" ] || [ ! -r "$logfile" ]; then
  echo "usage: $0 logfile" >&2
  exit 2
fi

# Keep the pattern list in a single grep -E expression.
# Avoid arrays, mapfile, process substitution, and [[ ]].
pattern='upstream timed out|connect\(\) failed|no live upstreams'

cluster=$(grep -E "$pattern" "$logfile" | tail -n 20 || true)
count=$(printf '%s\n' "$cluster" | grep -c . || true)

printf 'cluster_lines=%s\n' "$count"
if [ -n "$cluster" ]; then
  printf '%s\n' "$cluster"
fi

# Non-zero when the cluster is growing past a boring baseline.
if [ "$count" -ge 5 ]; then
  exit 1
fi
exit 0

Python caller that does not smuggle /bin/sh

# example caller — no shell=True, so argv does not pass through dash
import subprocess
import sys

def run_triage(logfile: str) -> int:
    proc = subprocess.run(
        ["/bin/sh", "./triage.sh", logfile],
        check=False,
        text=True,
        capture_output=True,
    )
    sys.stdout.write(proc.stdout)
    sys.stderr.write(proc.stderr)
    return proc.returncode

if __name__ == "__main__":
    raise SystemExit(run_triage(sys.argv[1]))

Notice the shebang on the shell file and the explicit /bin/sh on the Python side. If I need bash features later, I will write a bash shebang and invoke bash from Python. Hope is not a runtime contract, and a .sh suffix is not an interpreter you can trust.

A tiny test plan

I now run this checklist before I believe a helper on a second machine that I did not provision:

1. head -n 1 triage.sh shows a real shebang, not a blank line.
2. sh -n triage.sh parses without syntax errors.
3. bash -n triage.sh also parses, so I have not invented the opposite lock-in.
4. sh detect-shell.sh and bash detect-shell.sh both print, and I record the dialect.
5. python3 caller.py sample.log matches sh triage.sh sample.log on exit code and line count.
6. A fixture log with four matching lines exits 0; a fixture with five matching lines exits 1.

You can build the fixtures with printf and never wait on production rotation to prove the exit codes.

printf '%s\n' \
  "2026/09/16 10:01:02 [error] 17#17: *1 connect() failed" \
  "2026/09/16 10:01:03 [error] 17#17: *2 upstream timed out" \
  "2026/09/16 10:01:04 [error] 17#17: *3 no live upstreams" \
  "2026/09/16 10:01:05 [error] 17#17: *4 connect() failed" \
  "2026/09/16 10:01:06 [error] 17#17: *5 upstream timed out" \
  > /tmp/sample.error.log
sh -n triage.sh
sh triage.sh /tmp/sample.error.log; echo exit:$?

If the fixture with five matching lines still exits zero, the helper is lying even when the shell dialect is correct.

Decision table I wish I had taped to the monitor

• [[ expr ]] → [ expr ] or test expr, with quoted expansions
• array=( a b ) and "${array[*]}" → a single string or repeated arguments
• pipefail → check each stage, or accept that grep | tail hides grep's exit
• mapfile / <<< / <() → while IFS= read -r line; do ...; done
• echo -e → printf '%s\n'
• source file → . file
• == inside test → =
• grep -P → grep -E or a language you actually pinned
• shell=True in Python → an argv list plus an explicit interpreter

That table is not a moral lecture about bash, because bash is a fine contract when every layer names it. The failure mode is an implied contract, where the filename, the chat draft, and /bin/sh all mean different languages. Pin the interpreter in the shebang and in the caller, or stop pretending the helper is portable.

What I would repeat

I would still draft the first helper on the same machine that will run it later. Laptop-only optimism is how dialect bugs get a two-day head start before anyone reads stderr. I would still keep the Python watchdog tiny and let the shell helper own the log grammar. I would still insist on fixture files instead of tailing a live log while thinking I was testing.

I would also repeat the three-way invocation as a ritual: bash, sh, and the actual parent process. Interactive success is a demo you can screenshot. The parent process is the only runtime that counts.

What broke, and what I would not repeat

I would not paste a generated script without a shebang and then ask the server to guess the dialect. I would not use shell=True as a default just because a pipeline looked prettier in a draft. I would not spend the first evening collecting ulimit output when the parser had not accepted the file. I would not treat a remote environment as a unique snowflake until /bin/sh and the shebang have been printed.

Shared boxes do have real limits, but dialect bugs impersonate those limits with embarrassing accuracy. Forty-eight hours is a long time to learn that triage.sh is a filename, not a runtime guarantee. Print the interpreter first, then collect metrics, or you will debug the wrong layer twice.

Limitations, and who should skip this

This approach assumes you can choose POSIX sh, or that you can pin bash everywhere you execute the file. If your helper needs arrays, associative maps, or coproc, do not translate it into dash for sport. Pin a bash shebang, install bash on the image, and invoke bash from Python with an argv list.

This approach also assumes the log format is stable enough for grep -E to remain honest. If your errors are multiline JSON, a Python parser is the helper, and the shell should only exec it. The detector script does not prove correctness; it only proves you stopped lying about the dialect.

Skip this if you already pin the shell in an image and run sh -n in CI. You do not need a field-notes ritual for a contract that is already enforced by tests. Skip it if you cannot read the shebang on the machine that actually executes the file. Advice about /bin/sh is worthless as folklore that never touches the host.

Generated drafts will still emit bash by default, because most public snippets are bash, not dash. A remote server does not change that prior; it only changes whether you notice, and I noticed late.

If you borrow one check from this write-up, make it the three-way run of detect-shell.sh before you keep any generated helper. That is the whole lesson I still trust after watching /bin/sh quietly refuse my first draft. Have you checked what readlink -f /bin/sh prints on the box you are about to blame?