#!/usr/bin/env python3 -u Worked Under python3 and Failed as ./prune.py: What Linux Does With a Shebang Line

I gave a clip-pruning script the first line #!/usr/bin/env python3 -u, tested it with python3 prune.py clips, and it worked. Run as ./prune.py clips, the way a cron job or a systemd timer would start it, it failed with exit status 127 before a single line of Python ran.

The script was fine. The first line was not, and the way I tested it skipped the first line entirely.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it stores or prunes files. Every script ran on one Linux machine (kernel 6.8, aarch64) with GNU bash 5.1.16, GNU coreutils 8.32 and Python 3.10.12, against a scratch folder the setup script creates under /tmp.

What the tests showed:

• The kernel passes everything after the interpreter as one argument. -u -B arrived as a single string, space included.
• That breaks env. It went looking for a program literally named python3 -u.
• Two flags break the interpreter too, even with no env in the line. One flag works. Two flags joined into one word work.
• env -S fixes it by splitting the string itself.
• A line that is too long is cut silently. No error, no warning, a shorter argument.
• Windows line endings fail the same way, because the carriage return becomes part of the interpreter's name.
• Running the script through its interpreter hides every one of these. That is why they survive testing.

Setup

setup.sh makes the scratch folder, two empty clips, and showargs, a tiny Python program that prints exactly what it was handed. Using it as an interpreter shows what the kernel passes.

#!/bin/bash
# Builds a scratch folder with a tiny "show me my argv" interpreter.
set -eu
D=/tmp/shebang_demo
rm -rf "$D"; mkdir -p "$D/clips"
touch "$D/clips/clip_0001.mp4" "$D/clips/clip_0002.mp4"
cat > "$D/showargs" <<'PY'
#!/usr/bin/python3
import sys
for i, a in enumerate(sys.argv):
    print("argv[%d] = %r" % (i, a))
PY
chmod +x "$D/showargs"
uname -sr
bash --version | head -1
env --version | head -1
python3 --version

Linux 6.8.0-138-generic
GNU bash, version 5.1.16(1)-release (aarch64-unknown-linux-gnu)
env (GNU coreutils) 8.32
Python 3.10.12

Test one: what the kernel hands the interpreter

#!/bin/bash
# What the kernel actually hands the interpreter named on the #! line.
D=/tmp/shebang_demo
printf '#!%s -u -B\nprint("body never runs")\n' "$D/showargs" > "$D/job.py"
chmod +x "$D/job.py"
"$D/job.py" clips

Output:

argv[0] = '/tmp/shebang_demo/showargs'
argv[1] = '-u -B'
argv[2] = '/tmp/shebang_demo/job.py'
argv[3] = 'clips'

The first argument after the program's own path is '-u -B', one string. The execve(2) manual page says so directly: "On Linux, the entire string following the interpreter name is passed as a single argument to the interpreter, and this string can include white space." The kernel does no word splitting on that line. Whatever program you name gets one argument and has to cope with the space itself.

Note also that the script path arrives as an argument after it, and the name the caller used for the program is gone. The same page: "Note that there is no way to get the argv[0] that was passed to the execve() call."

Test two: the common env line

#!/bin/bash
# The common shebang: env plus an interpreter flag.
D=/tmp/shebang_demo
cat > "$D/prune.py" <<'PY'
#!/usr/bin/env python3 -u
import sys, glob
print(len(glob.glob(sys.argv[1] + "/*.mp4")), "clips")
PY
chmod +x "$D/prune.py"
"$D/prune.py" "$D/clips"
echo "exit status: $?"
python3 "$D/prune.py" "$D/clips"
echo "exit status: $?"

Output:

/usr/bin/env: ‘python3 -u’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines
exit status: 127
2 clips
exit status: 0

env received python3 -u as one argument and searched PATH for a program with that whole name, space and all. There is none, so it exited 127, which the GNU env(1) manual page defines as "127 if COMMAND cannot be found". The second line of output is GNU env recognising the pattern and suggesting the fix. Older or non-GNU builds may print only the first line.

The last two lines are the trap. python3 prune.py worked, because the shell started python3 directly and Python treats the #! line as a comment. I tested the script the one way that skips the line that was broken.

Test three: no env, two flags

Dropping env and naming the interpreter by absolute path is a common suggestion. It moves the failure rather than removing it.

#!/bin/bash
# Absolute interpreter path, no env: one flag, two flags, two flags joined.
D=/tmp/shebang_demo
for line in '#!/usr/bin/python3 -u' '#!/usr/bin/python3 -u -B' '#!/usr/bin/python3 -uB' \
            '#!/bin/bash -e' '#!/bin/bash -e -u' '#!/bin/bash -eu'; do
  case "$line" in
    *python3*) body='print("ran")' ;;
    *)         body='echo ran' ;;
  esac
  printf '%s\n%s\n' "$line" "$body" > "$D/t"
  chmod +x "$D/t"
  out=$("$D/t" 2>&1); rc=$?
  printf '%-28s exit %-3s %s\n' "$line" "$rc" "$(printf '%s' "$out" | head -1)"
done

Output:

#!/usr/bin/python3 -u        exit 0   ran
#!/usr/bin/python3 -u -B     exit 2   Unknown option: - 
#!/usr/bin/python3 -uB       exit 0   ran
#!/bin/bash -e               exit 0   ran
#!/bin/bash -e -u            exit 1   /bin/bash: - : invalid option
#!/bin/bash -eu              exit 0   ran

One flag is fine, because one flag is one word. Two flags separated by a space reach Python as the single argument -u -B, and Python reads the space as an option character it does not know. Bash fails the same way for -e -u. Joining the flags into one word, -uB or -eu, works because it really is a single argument.

The manual page's own advice matches what this shows: "For portable use, optional-arg should either be absent, or be specified as a single word (i.e., it should not contain white space)".

Test four: the env -S fix

#!/bin/bash
# The fix: env -S splits the one string the kernel passed it.
D=/tmp/shebang_demo
sed -i '1s|.*|#!/usr/bin/env -S python3 -u|' "$D/prune.py"
head -1 "$D/prune.py"
"$D/prune.py" "$D/clips"
echo "exit status: $?"
printf '#!/usr/bin/env -S %s -u -B\n' "$D/showargs" > "$D/job2.py"
chmod +x "$D/job2.py"
"$D/job2.py" clips

Output:

#!/usr/bin/env -S python3 -u
2 clips
exit status: 0
argv[0] = '/tmp/shebang_demo/showargs'
argv[1] = '-u'
argv[2] = '-B'
argv[3] = '/tmp/shebang_demo/job2.py'
argv[4] = 'clips'

The kernel still passes one string. The difference is that env now splits it. The env(1) page describes the option as "process and split S into separate arguments; used to pass multiple arguments on shebang lines". showargs confirms it: -u and -B arrived as two arguments.

Not every env has -S. I only tested GNU coreutils 8.32, so check env --help on the machine that will run the job, not only on yours.

Test five: a line that is too long

#!/bin/bash
# A #! line longer than the kernel reads. Count what survives.
D=/tmp/shebang_demo
arg="--tag=$(printf 'x%.0s' $(seq 1 300))"
printf '#!%s %s\n' "$D/showargs" "$arg" > "$D/long.py"
chmod +x "$D/long.py"
echo "characters after #! in the file: $(head -1 "$D/long.py" | tail -c +3 | tr -d '\n' | wc -c)"
"$D/long.py" | python3 -c '
import sys
for line in sys.stdin:
    if line.startswith("argv[1]"):
        val = eval(line.split(" = ", 1)[1])
        print("argv[1] the interpreter received: %d characters" % len(val))
    else:
        print(line, end="")
'
echo "exit status: ${PIPESTATUS[0]}"

Output:

characters after #! in the file: 333
argv[0] = '/tmp/shebang_demo/showargs'
argv[1] the interpreter received: 226 characters
argv[2] = '/tmp/shebang_demo/long.py'
exit status: 0

The file had 333 characters after #!. The interpreter path and its trailing space take 27 of them, and the argument that arrived was 226 characters, so 253 characters survived. Exit status 0. Nothing reported the cut.

The manual page says "characters beyond the limit are ignored", and gives the limit: "Since Linux 5.1, the limit is 255 characters."

My count came out two short of that. I have not traced where the difference goes, and I would not build anything that sits within a few characters of either number. The useful point is the one I could measure: a long line is shortened, not rejected. A long interpreter path, such as one inside a deep virtual environment folder, can lose its tail and point at a program that does not exist.

Test six: Windows line endings

A script edited on Windows and copied to a Linux box or a Raspberry Pi often keeps its carriage returns.

#!/bin/bash
# The same script saved with Windows line endings.
D=/tmp/shebang_demo
printf '#!/bin/bash\r\necho ran\r\n' > "$D/crlf.sh"
chmod +x "$D/crlf.sh"
"$D/crlf.sh"
echo "exit status: $?"
head -1 "$D/crlf.sh" | od -c | head -1
bash "$D/crlf.sh" > "$D/crlf.txt"
echo "exit status: $?"
od -c "$D/crlf.txt" | head -1

Output:

crlf.sh: /tmp/shebang_demo/crlf.sh: /bin/bash^M: bad interpreter: No such file or directory
exit status: 126
0000000   #   !   /   b   i   n   /   b   a   s   h  \r  \n
exit status: 0
0000000   r   a   n  \r  \n

Run directly, it failed with exit 126 and bad interpreter. The od line shows why: the first line ends in \r before the newline, so the kernel looked for an interpreter named /bin/bash followed by a carriage return. Bash prints that carriage return as ^M.

Run as bash crlf.sh, it exited 0, and the last line shows the cost of that: the output was ran followed by a stray \r. The script ran and quietly put a carriage return into everything it printed. If that output were a file name or a value passed to another command, the stray byte would go with it.

What to do instead

• Keep the #! line to the interpreter plus at most one word: #!/usr/bin/python3 -u or #!/bin/bash -eu.
• If you need env and more than one word, use #!/usr/bin/env -S python3 -u, and confirm the target system's env has -S.
• Set options inside the script where you can. set -eu on the second line of a bash script does not depend on the shebang at all.
• Test scripts the way they will be started: ./script, not python3 script or bash script.
• Check new scripts for carriage returns with head -1 script | od -c before deploying them.
• Treat exit status 126 and 127 from a scheduled job as a start failure, not a script failure. Nothing inside the script ran.

Things I did not test

• macOS and the BSDs. The manual page says behaviour differs: "Some systems use the first white space to terminate optional-arg." Only Linux was tested here.
• Kernels before 5.1, where the documented limit is 127 characters.
• Android. Nothing here ran on a phone or through adb shell.
• Other env builds, such as BusyBox, which may not support -S.

Sources. Quotes are from the Linux execve(2) manual page (man-pages 6.19) and the GNU env(1) manual page (coreutils 9.11). Results come from the scripts above, each run once, with output pasted unedited.

If you want an old phone to record clips in the background so there is a folder worth pruning, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

Not one line of the script ran; the line above it decided that.