I Blamed bash for 48 Hours. The Generated Script Still Ended Lines With CR.

Have you ever watched a generated deploy script fail on a remote box while every local check stayed green? I just burned forty-eight hours on that mismatch, and the chat log kept insisting the file was fine. The script had a shebang, a plus-x bit, and a smoke test that looked copied from a tutorial. Why did the remote kernel still print an interpreter error that made no sense?

This is a field-notes writeup, not a victory lap, and I will stay with the bytes that actually moved. I will walk through what I tried, what broke, and the small checker I would run again tomorrow morning. The useful part survives even if you never touch the assistant that drafted the original shell script.

Hour 0–8: I trusted the generated file

I asked a coding assistant for a tiny helper that would sync an app, install dependencies, and curl health. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to draft that helper, then I planned to run it on the free server option. I wanted a second machine that was not my laptop, because laptops hide a surprising number of text-file sins.

The first draft looked reasonable enough that I almost skipped every mechanical check I usually run. It had a shebang, set -euo pipefail, and a curl invocation that failed loudly on HTTP errors. Does a tidy snippet still count as evidence, or is it only a preview of bytes you have not measured?

#!/usr/bin/env bash
set -euo pipefail
APP_DIR="${APP_DIR:-$HOME/demoapp}"
cd "$APP_DIR"
python3 -m pip install -r requirements.txt
python3 -m app &
sleep 2
curl -fsS "http://127.0.0.1:8000/health"

Does that snippet look dangerous to you yet, or does it still read like a tutorial you would paste? It should worry you, because the background job, the hardcoded port, and the two-second sleep are stacked assumptions. I still copied it into deploy.sh, ran chmod +x, and told myself the remote kernel would behave the same.

Hour 8–20: I chased PATH, python, and permissions

The remote shell rejected the script with a message I misread as a PATH problem. Have you stared at bad interpreter so long that you start reinstalling bash packages for no reason? I did, and I also reinstalled python3, which fixed nothing on the box. The error wrapped in the terminal, so the carriage return never showed up as a visible character.

What I tried, in order, before I looked at raw bytes:

1. which bash plus ls -l /usr/bin/env, expecting a missing interpreter package.
2. chmod +x deploy.sh again, then namei -l ./deploy.sh for a permission story.
3. Switching the shebang from /usr/bin/env bash to a hardcoded /bin/bash path.
4. Running bash deploy.sh so the kernel would not consult the shebang line.
5. Comparing login shells against non-login SSH sessions, which wasted an entire evening.

Item four should have ended the incident, and it did not, because I still scp'd the dirty file. bash deploy.sh worked in one shell and failed in another, which I blamed on profile scripts. Does that sound familiar if you live inside SSH sessions and keep exporting PATH by hand?

The commands that finally made the bug visible were boring, and I wish they had been hour-one muscle memory:

file deploy.sh
head -n 1 deploy.sh | cat -A
head -n 1 deploy.sh | od -An -tx1
git diff --check deploy.sh

file printed ASCII text, with CRLF line terminators, which is the whole incident in one clause. The first line's bytes ended with 0d 0a, not a lone 0a. The kernel was looking for an interpreter named /usr/bin/env bash\r. Of course that path was missing.

What actually broke

The model wrote the script in a buffer that used Windows line endings, and I saved it without asking Git to normalize anything. My laptop could hide that, because some local tools tolerate CR and LF together in the same file. The Linux box did not tolerate it, and the error string wrapped in a way that hid the \r. Have you noticed how many editors preview a file instead of showing od?

I had also let the assistant "verify" the file by reading it back as text in the chat transcript. Text readers strip or normalize carriage returns all the time, so the transcript could show a perfect shebang while the bytes on disk stayed wrong. A rendered buffer is not a test, and a green paraphrase is not a checksum. Would you ship a binary after reading a hex dump that someone else summarized in prose?

A second failure showed up after I finally fixed the endings and copied the file again. The script backgrounded python3 -m app, slept two seconds, then curled localhost as if warmup were a constant. Locally the app was already imported and warm from earlier runs. On the free server, importing the app module took longer than that sleep, so curl hit a closed port and I blamed networking for another night.

The line-ending bug and the sleep race stacked, which is why the incident lasted two days instead of twenty minutes. I was debugging two failures as if they were one story. Field notes only started helping when I separated "the kernel cannot see bash" from "the process is not listening yet."

The artifact: fail the file before SSH

I do not want another forty-eight hour loop on the same class of failure, so the checker below is ordinary Python. It reads bytes, not a preview, and it fails the deploy file before SSH ever runs it. Treat it as a local gate, not as a claim about any model's accuracy. Label: this is a script I would keep in the repo and run on a working tree before copy.

#!/usr/bin/env python3
"""Fail a deploy script that still contains CR or a fragile shebang."""
from __future__ import annotations

import stat
import sys
from pathlib import Path

DEPLOY = Path("deploy.sh")
CR = b"\r"


def fail(message: str) -> None:
    print(f"FAIL: {message}", file=sys.stderr)
    raise SystemExit(1)


def main() -> None:
    if not DEPLOY.is_file():
        fail(f"{DEPLOY} is missing")

    data = DEPLOY.read_bytes()
    if not data:
        fail("deploy.sh is empty")
    if CR in data:
        fail("deploy.sh contains CR bytes; convert to LF before SSH")

    first, sep, _rest = data.partition(b"\n")
    if sep != b"\n":
        fail("deploy.sh has no LF newline")
    if not first.startswith(b"#!"):
        fail("deploy.sh has no shebang on line 1")

    # /usr/bin/env bash is fine; extra shebang flags are not portable.
    parts = first[2:].strip().split()
    if first.startswith(b"#!/usr/bin/env") and len(parts) > 2:
        fail("shebang passes extra args that some kernels ignore")

    mode = DEPLOY.stat().st_mode
    if mode & (stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH) == 0:
        fail("deploy.sh is not executable")

    print("PASS: deploy.sh has LF endings, a shebang, and an executable bit")


if __name__ == "__main__":
    main()

I also keep a tiny remote smoke test that does not use sleep as if it were a readiness probe. The loop waits on the TCP port, then checks a JSON health body, and it gives up with a non-zero status. Label: /dev/tcp is a bash extension; swap it for socket.create_connection if your remote shell is not bash.

#!/usr/bin/env bash
# remote_smoke.sh — wait for TCP, then require a JSON health body.
set -euo pipefail
host="${1:-127.0.0.1}"
port="${2:-8000}"
deadline=$((SECONDS + 30))

while ! bash -c "echo >/dev/tcp/${host}/${port}" 2>/dev/null; do
  if (( SECONDS >= deadline )); then
    echo "port ${port} never opened" >&2
    exit 1
  fi
  sleep 0.2
done

body="$(curl -fsS "http://${host}:${port}/health")"
python3 - "$body" <<'PY'
import json, sys
data = json.loads(sys.argv[1])
raise SystemExit(data.get("status") != "ok")
PY

If you do not want /dev/tcp, replace the wait loop with this labeled Python probe instead of inventing another sleep:

python3 - "$host" "$port" <<'PY'
import socket, sys, time
host, port = sys.argv[1], int(sys.argv[2])
deadline = time.time() + 30
while time.time() < deadline:
    try:
        with socket.create_connection((host, port), timeout=0.5):
            raise SystemExit(0)
    except OSError:
        time.sleep(0.2)
raise SystemExit("port never opened")
PY

The point is the same in both shapes: wait for the real listener, then parse a real body. A two-second sleep is not a probe, and a chat summary of curl is not a probe either.

A tiny test around the checker

I want the gate itself to fail in a test, not only in production SSH. The snippet below writes a dirty file with CRLF, runs the checker, and expects a non-zero exit. Label: this is a reproducible local test, not a benchmark and not a claim about remote hardware.

# test_check_deploy.py
from pathlib import Path
import runpy
import sys

def test_crlf_is_rejected(tmp_path, monkeypatch):
    script = tmp_path / "deploy.sh"
    script.write_bytes(b"#!/usr/bin/env bash\r\necho hi\r\n")
    script.chmod(0o755)
    monkeypatch.chdir(tmp_path)
    monkeypatch.setattr(sys, "argv", ["check_deploy.py"])
    try:
        runpy.run_path("check_deploy.py", run_name="__main__")
    except SystemExit as exc:
        assert exc.code == 1
        return
    raise AssertionError("CRLF deploy.sh should not pass")

Put check_deploy.py on PYTHONPATH or in the same directory, and run python3 -m pytest test_check_deploy.py. If that test ever goes green on a CRLF file, the gate is lying and you should not copy anything.

Commands I would repeat

Here is the short list I wish I had run in hour one, not hour thirty, before any remote debugging story:

• git config --get core.autocrlf and git config --get core.eol
• printf '*.sh text eol=lf\n' >> .gitattributes
• file deploy.sh requirements.txt
• python3 check_deploy.py
• ssh user@host 'file ~/deploy.sh; od -An -tx1 -N 16 ~/deploy.sh'

Should .gitattributes be optional on a mixed OS team? Not if a model or an editor can reintroduce CRLF on the next generation. I now keep *.sh text eol=lf in the repo so Git rewrites the bytes on checkout. That does not fix a file you scp from a dirty working tree, so the Python checker still runs before copy.

I also stopped asking the chat to "make sure the script is Unix." That sentence has no bytes in it. If the assistant rewrites deploy.sh, I run file and the checker again, because generation is not idempotent with line endings.

Decision table

Situation	Use a free model plus a free second server this way?	Why
Learning a deploy path, no secrets	Yes, with the checker and a real smoke test	You get a second kernel and a second filesystem
Production traffic or customer data	No	A free server is the wrong isolation boundary
The chat says tests passed with no artifact	No, not until you have a log or an exit code	Summaries are not evidence
Shell scripts generated on a mixed OS team	Yes, only if CRLF is gated	This exact class of bug is cheap to catch
You cannot SSH and read raw bytes	No	You will debug previews instead of files

What I would repeat, and what I would not

I would still use a free model to draft the boring glue, because glue is where I make tired mistakes after midnight. I would still use a free server as a second machine, because my laptop hides Windows-tolerant behavior that Linux will not forgive. I would not paste the generated script onto the server until check_deploy.py and file both agree. I would not let sleep 2 stand in for readiness on a box I do not control.

Would I trust a rendered diff in the chat again after this incident? Only as a hint that I still have to prove with od. I want file, a health body that parses as JSON, and a non-zero exit when either check fails. The model can write the first draft of the glue. The remote kernel still decides whether the shebang is a path or a path plus CR.

Limitations

This workflow does not replace CI, and it does not prove the application is correct under load. It only catches a cluster of deploy-file bugs that wasted two days: line endings, shebang shape, the executable bit, and a sleep-shaped race. It will not catch unpinned dependency drift, an app that binds only to IPv6, or a 200 response with the wrong JSON schema beyond one assertion.

Who should skip this? Anyone shipping secrets to a shared box should skip the free-server half entirely and use an isolated environment they actually control. Anyone without SSH or another way to read raw bytes should skip the claim that a remote run succeeded. Anyone hoping the model will self-certify a deploy should skip the whole pattern, because that was the original failure mode.

If you already generate glue scripts the same way I did, keep the checker and the TCP wait. That is the part I would repeat on the next forty-eight hour incident, before I blame bash again.

Comments

[P O]

CRLF from generated scripts is a classic. Before you rewrite half the script, run:

file script.sh
od -c script.sh | head

If you see \r, dos2unix script.sh (or sed -i 's/\r$//' script.sh) usually fixes the mysterious bash errors in one shot. Editors and AI tools on Windows love leaving those behind.