The Preflight Exited 0 for 48 Hours. The Server's sh Was dash.

Have you ever watched a deploy script print success while the binary it built was never created? I have, and the silence lasted long enough to ruin two nights of otherwise boring glue work. The Python service looked healthy, the preflight returned zero, and the artifact directory still stayed empty. What kind of passing check leaves you with nothing to run when Monday morning arrives?

Field notes, not a victory lap

This is a reconstructed 48-hour log of a class of bugs I keep stepping on, not a benchmark report. Every command below is a reproduction you can run on a Debian-like host and on a Mac. I am not claiming production traffic, customer impact, or a timed bake-off against any other tool. If a number is not in a command output here, I do not have that number.

I needed a Linux shell that was not my laptop, plus a model that could draft the glue without a procurement thread. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to sketch the preflight script, then the free server option to run the same files on Linux. My laptop kept lying to me because /bin/sh there is not the same program.

Hour 0–6: I blamed the HTTP client

The generated preflight compiled a tiny helper, hashed it, and then asked Python to boot the service. I wired subprocess.run(..., check=True) around the script and treated a zero exit as gospel. When the helper was missing, the Python process still started, and I went hunting in retries, DNS, and TLS. Does that sound familiar if you have ever trusted a wrapper more than a file listing?

Here is the shape of the first script, and you should treat it as a labeled reproduction. I did not copy this from a live repository, and I want that limitation to stay visible. Paste it into an empty directory if you want the same bruise I got.

#!/bin/sh
set -e
mkdir -p dist
gcc -O2 helper.c -o dist/helper | tail -n 1
echo "preflight ok"

# preflight_runner.py — reproduction, not production telemetry
import subprocess
import sys
from pathlib import Path

def run_preflight(script: str = "./preflight.sh") -> None:
    completed = subprocess.run(
        ["/bin/sh", script],
        check=True,
        text=True,
    )
    print(f"wrapper saw returncode={completed.returncode}")
    if not Path("dist/helper").exists():
        raise SystemExit("helper missing after a 'successful' preflight")

if __name__ == "__main__":
    run_preflight(sys.argv[1] if len(sys.argv) > 1 else "./preflight.sh")

The gcc command can fail, the tail command can still succeed, and set -e will not save you here. On my laptop I sometimes launch scripts with bash, so the muscle memory of pipefail hid in my interactive profile. Did the non-interactive server inherit that profile, or did it start dash with a clean POSIX personality? It started dash with a clean POSIX personality, and that single fact explains the rest of this log.

Hour 6–18: more logs, same zero

I did what everyone does when a wrapper looks honest and the service still misbehaves in boring ways. I added echo lines, printed pwd, and dumped env into notes that explained almost nothing useful. The Linux host kept returning zero from /bin/sh, and my Mac kept matching that lie. Which process was I actually debugging in those hours, the compiler or the shell itself?

What I tried, in order

1. Restarted the Python process and blamed HTTP timeouts for a helper binary that was never on disk.
2. Printed sys.executable and os.getcwd() from the runner, which were both completely fine.
3. Ran bash preflight.sh by hand, finally saw gcc fail, then kept using /bin/sh in code.
4. Checked ls -l dist/ only after the service had already bound a local port.
5. Asked the model to make the script stricter, and received more echo statements around the same pipe.

The fifth step is the embarrassing one because it felt like progress while changing nothing about process identity. A model will happily decorate a pipeline without changing the process that consumes the pipeline at runtime. Have you noticed how often add-more-logging is a polite way to avoid naming the shell? I noticed it only after the server and the laptop disagreed about what /bin/sh meant.

Hour 18–30: the server spoke dash

On Debian and Ubuntu images, /bin/sh is dash, the Debian Almquist shell, and it is not bash. Dash does not implement [[, process substitution, or set -o pipefail the way bash users expect. My Mac still points /bin/sh at bash in many setups, so the same shebang is two different languages. Why do we keep writing POSIX shebangs and then pasting bash snippets from muscle memory without checking?

I confirmed it with boring commands on the Linux box, and none of them required a special agent. You can run the same block on your host before you trust any generated preflight script again. If readlink prints dash, your bash-only options are fiction even when the model emits them.

ls -l /bin/sh
# GNU coreutils; on macOS this flag is often missing, so keep ls -l above.
readlink -f /bin/sh || true
python3 -c 'import os; print(os.path.realpath("/bin/sh"))'
/bin/sh -c 'echo "sh=$0"; set -o pipefail'
/bin/bash -c 'echo "bash=$0"; set -o pipefail; echo pipefail_ok'
command -v gcc || echo "gcc missing"

Dash rejects set -o pipefail with an error. Bash accepts it and then makes the pipeline's status the last non-zero command. If your image has no compiler, gcc | tail still exits zero under dash because tail succeeded. That is not flaky hardware. That is POSIX.

On the laptop, GNU readlink -f is not a portable spell, so I printed ls -l /bin/sh instead. A Mac can look “green” because /bin/sh is bash in POSIX clothes, not because my script was portable. The second host is the experiment. Without it I would still be decorating echo lines.

A decision table I wish I had drawn at hour two

| How you invoke it | Which shell | pipefail | gcc missing, gcc \| tail |
| --- | --- | --- | --- |
| subprocess.run(["/bin/sh", script]) on Debian | dash | not available | exit 0, no binary |
| subprocess.run(["/bin/bash", script]) without -o pipefail | bash | off | exit 0, no binary |
| bash -o pipefail script or set -o pipefail inside | bash | on | non-zero, wrapper raises |
| Shebang #!/bin/bash plus set -euo pipefail | bash | on | non-zero if gcc fails |
| Shebang #!/bin/sh plus [[ -x gcc ]] | dash | n/a | syntax error, or ignored |

The table is the artifact I now keep next to any script a model proposes during review. If the row does not name the binary, I do not trust the exit code that comes back. Should a coding model know this POSIX split, or will it emit [[ anyway under time pressure? Sometimes it mentions POSIX shells in the commentary and still writes bash in the file.

Hour 30–48: I stopped asking the shell to be bash

The fix was not a smarter retry loop and not a longer prompt to the same model. I pinned the shell, killed the pipeline, and made Python own the existence check after the child. The model can still draft the comments, but the runner must refuse a green exit when files are missing. Would I ship a service that never proved dist/helper exists on disk after preflight returns zero?

#!/usr/bin/env bash
set -euo pipefail

mkdir -p dist
if ! command -v gcc >/dev/null 2>&1; then
  echo "gcc is not on PATH" >&2
  exit 127
fi

gcc -O2 helper.c -o dist/helper
test -x dist/helper
echo "preflight ok"

# test_preflight.py — run with: python -m pytest test_preflight.py -q
from pathlib import Path
import subprocess
import textwrap

def write_script(tmp_path: Path, body: str, name: str = "preflight.sh") -> Path:
    script = tmp_path / name
    script.write_text(textwrap.dedent(body), encoding="utf-8")
    script.chmod(0o755)
    return script

def test_pipeline_hides_gcc_failure(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    script = write_script(
        tmp_path,
        """\
        #!/bin/sh
        set -e
        mkdir -p dist
        false | tail -n 1
        echo preflight_ok
        """,
    )
    completed = subprocess.run(["/bin/sh", str(script)])
    assert completed.returncode == 0  # documents the trap

def test_bash_pipefail_surfaces_failure(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    script = write_script(
        tmp_path,
        """\
        #!/usr/bin/env bash
        set -euo pipefail
        false | tail -n 1
        echo preflight_ok
        """,
    )
    completed = subprocess.run(["/usr/bin/env", "bash", str(script)])
    assert completed.returncode != 0

The first test is supposed to pass, and that still feels wrong when you read it out loud. It is a characterization test for the bug, not a celebration of dash or of sloppy pipelines. The second test is the contract I want the Linux host to enforce before Python starts listening at all. If bash is not installed on a slim image, the second test fails loudly, which is the entire point.

What I would repeat

I would still let a model draft the first version of a preflight script, because the boring structure is cheap. I would not let that draft define the process binary, and I would not trust set -e across a pipe. I would copy the decision table into the pull request so the next reviewer does not rediscover dash. That is the part I will repeat even when the host is a machine I already pay for.

A short checklist now sits above any subprocess call that I generate, review, or paste from a model. The items look obvious after the incident, which is why they belong on the page and not in my head. I still skip them when I am tired, so they are written where the runner lives.

• Name the interpreter in the argv list, and do not hide it behind shell=True.
• Avoid pipelines when a single command can fail closed on its own.
• Assert the artifact with Path.exists() in Python after the child returns.
• Run the same files under /bin/sh and under bash before you call the job green.
• Read ls -l /bin/sh on every new host, including ones you did not provision by hand.

Would I skip the extra Linux host if I already had a Debian container on the laptop this week? Yes, without hesitation, because the lesson is the shell dialect, not the brand of the host. The second machine only mattered because it had a different /bin/sh than my laptop. That difference is the whole experiment, and it does not require a GPU or a special image beyond Linux.

Limitations, and who should not bother

This workflow is for people who wrap native tools from Python and ship those wrappers onto Debian-like hosts. It will not help you if your build is pure Python with no subprocess and no native helper binary. It will waste time if you only ever run bash on identical developer laptops that never see dash. Windows images do not have dash, and BusyBox sh is yet another dialect, so the table is not universal law.

I also did not measure model quality, token cost, or server hardware, because none of those numbers would have found pipefail. Free model access can emit a plausible script that is still wrong under a POSIX /bin/sh. A free server can reproduce that wrongness only if you invoke /bin/sh the way production will invoke it. If your production entrypoint is already a pinned bash with pipefail, you already solved the interesting part.

The other limitation is social, and it shows up in code review rather than in the shell man page. Characterization tests that assert returncode equals zero for a known bug look like mistakes during hurried review. Label them in the assertion message so future me does not “fix” the test by switching shells. I write a short comment on that assertion so the lesson survives a cleanup pass.

What broke, in one paragraph

The model wrote a POSIX shebang and a bash brain, and those two things are not the same language. My laptop sometimes honored the brain, especially when I typed bash without noticing the shebang at the top. The Linux /bin/sh honored the shebang, ignored pipefail, and let tail launder a failed compiler invocation. Python's check=True then laundered that zero into a running service with no helper sitting on disk.

If you already have a Debian shell, you do not need another account to learn this particular failure. Run the two tests, then read ls -l /bin/sh before you trust the next generated wrapper. Pin the interpreter in the argv list, and stop arguing with dash about bash features it will never grow. That is the whole field note I wanted from those 48 hours of green exits.