One Apostrophe Stopped xargs Halfway Through a Checksum Manifest: Testing find | xargs Against Odd Filenames

Point find | xargs sha256sum at a folder of five camera clips and it can write a manifest with one line in it, print a single complaint about a quote, and exit 123, a code that reads like an ordinary command failure rather than a job that stopped reading its input partway through. That is what I got on a test folder whose names include a space, an apostrophe and a newline. I then ran four alternatives against the same folder to see which ones produce a complete manifest and which ones fail quietly.

An earlier post here, on why rm clips/*.mp4 said Argument list too long, ended by noting that piping names into xargs has its own filename hazards. This is that post.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it names or stores files. Everything here ran on one Linux machine (kernel 6.8, bash 5.1.16, GNU findutils 4.8.0, GNU coreutils 8.32), against files the scripts create themselves.

What the tests showed:

• find | xargs sha256sum hashed one clip out of five. A space split one name in two, and an apostrophe made xargs stop reading altogether.
• The exit status depended on which failure happened. With the split name in the folder it was 123. With only the apostrophe it was 1. A check for one specific code misses the other.
• find -print0 | xargs -0 hashed all five, including a name with a newline in it, and sha256sum -c verified the result.
• xargs -d '\n' fixed the space and the apostrophe but still broke on the newline.
• On a day with no clips, plain xargs -0 sha256sum wrote a line to the manifest anyway: the hash of empty input, with exit status 0. Adding -r stopped it.

The test folder

Five one-byte files, each named to test something different: a plain name, a space, an apostrophe, a plain name that sorts after the apostrophe, and a newline.

#!/bin/bash
# Build a test folder: one plain name, one with a space, one with an apostrophe,
# one ordinary name AFTER the apostrophe in sort order, and one with a newline.
rm -rf clips && mkdir clips
printf 'a' > "clips/0001_garage.mp4"
printf 'b' > "clips/0002_porch 0700.mp4"
printf 'c' > "clips/0003_dad's driveway.mp4"
printf 'd' > "clips/0004_side_gate.mp4"
printf 'e' > "clips/0005_back"$'\n'"yard.mp4"
ls -1b clips

ls -1b prints the names with escapes, so the space and the newline are visible:

0001_garage.mp4
0002_porch\ 0700.mp4
0003_dad's\ driveway.mp4
0004_side_gate.mp4
0005_back\nyard.mp4

Recording software tends to write plain names, so odd ones usually come from people: a clip renamed by hand to say where it was shot, a folder synced from a phone with a gallery app that keeps titles, or a file pulled in from someone else's export. One such name in a folder of thousands is enough.

The manifest job as first written

#!/bin/bash
# The manifest job as first written: find piped straight into xargs.
find clips -name '*.mp4' | sort | xargs sha256sum > manifest.txt
echo "xargs exit: $?"
echo "manifest lines: $(wc -l < manifest.txt)"
cat manifest.txt

Output:

xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
sha256sum: clips/0002_porch: No such file or directory
sha256sum: 0700.mp4: No such file or directory
xargs exit: 123
manifest lines: 1
ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb  clips/0001_garage.mp4

Two different things went wrong here.

The space is the expected one. The manual page says xargs reads items "delimited by blanks (which can be protected with double or single quotes or a backslash) or newlines", so 0002_porch 0700.mp4 arrived at sha256sum as two arguments, neither of which exists.

The apostrophe is worse. Because quotes are part of the syntax xargs parses, the apostrophe in 0003_dad's driveway.mp4 opened a quoted string that did not close. xargs reported the unmatched quote and stopped reading. Clips 0004 and 0005 were not passed to anything. They are missing from the manifest without a line of their own in the error output.

The manual page states the general problem directly: "Because Unix filenames can contain blanks and newlines, this default behaviour is often problematic". Apostrophes are not mentioned in that sentence, but the quoting rule above is what makes them fatal.

Exit status 123 or 1, depending

The first run exited 123. The exit status table gives 123 "if any invocation of the command exited with status other than 0 or 255", and here sha256sum had failed on the two halves of the split name. The quote problem was also present but did not change the code.

To see what xargs reports when the quote is the sole problem, I ran the same pipeline on a folder whose only odd name has an apostrophe:

#!/bin/bash
# Same job, but the only odd name is the apostrophe one.
rm -rf q && mkdir q
printf 'a' > "q/0001_garage.mp4"
printf 'c' > "q/0003_dads_driveway's.mp4"
printf 'd' > "q/0004_side_gate.mp4"
find q -name '*.mp4' | sort | xargs sha256sum > m.txt
echo "xargs exit: $?"
cat m.txt

xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
xargs exit: 1
ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb  q/0001_garage.mp4

Exit status 1, which the manual describes as some other error. The manifest again holds only the clip that came before the apostrophe.

So a wrapper script that alerts on 123 would have caught the first run and missed the second. Any non-zero status from this pipeline means the manifest is incomplete, and the job should treat it that way rather than matching a particular number.

NUL on both ends

The fix the manual recommends is -0, under which "the quotes and backslash are not special (every character is taken literally)". It only works if the producer also separates names with NUL bytes, so find gets -print0 and the sort in the middle gets -z:

#!/bin/bash
# NUL on both ends of the pipe: find -print0, sort -z, xargs -0.
find clips -name '*.mp4' -print0 | sort -z | xargs -0 sha256sum > manifest.txt
echo "xargs exit: $?"
echo "manifest lines: $(wc -l < manifest.txt)"
cat manifest.txt

xargs exit: 0
manifest lines: 5
ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb  clips/0001_garage.mp4
3e23e8160039594a33894f6564e1b1348bbd7a0088d42c4acb73eeaed59c009d  clips/0002_porch 0700.mp4
2e7d2c03a9507ae265ecf5b5356885a53393a2029d241394997265a1a25aefc6  clips/0003_dad's driveway.mp4
18ac3e7343f016890c510e93f935261169d9e3f565436429830faf0934f4f8e4  clips/0004_side_gate.mp4
\3f79bb7b435b05321651daefd374cdc681dc06faa65e374e38337b88ca046dea  clips/0005_back\nyard.mp4

All five clips are present. The last line is worth a closer look: sha256sum wrote the newline inside the filename as \n and put a backslash at the start of the line to mark that the name is escaped. That keeps the manifest one line per file, which matters for anything that reads it with line-based tools.

The escaped line also has to survive the trip back, so I verified the manifest with sha256sum itself:

#!/bin/bash
# Does the manifest written by fixed.sh verify, including the escaped newline line?
sha256sum -c manifest.txt
echo "check exit: $?"

clips/0001_garage.mp4: OK
clips/0002_porch 0700.mp4: OK
clips/0003_dad's driveway.mp4: OK
clips/0004_side_gate.mp4: OK
\clips/0005_back\nyard.mp4: OK
check exit: 0

All five verified, including the escaped one.

The middle ground: newline as the delimiter

GNU xargs also accepts -d '\n', which splits only on newlines and turns off quote handling. It is tempting when the producer cannot emit NUL bytes:

#!/bin/bash
# Middle ground: newline-delimited, quotes no longer special.
find clips -name '*.mp4' | sort | xargs -d '\n' sha256sum > manifest_d.txt
echo "xargs exit: $?"
echo "manifest lines: $(wc -l < manifest_d.txt)"

sha256sum: clips/0005_back: No such file or directory
sha256sum: yard.mp4: No such file or directory
xargs exit: 123
manifest lines: 4

The space and the apostrophe went through. The newline clip was split in two, exactly as the space was in the first run. If every name in the folder comes from software you control, this may be enough. If people can rename files, it is not.

The empty day

A camera that was unplugged all day leaves an empty folder. I expected an empty manifest:

#!/bin/bash
# A day with no clips. What lands in the manifest?
rm -rf none && mkdir none
find none -name '*.mp4' -print0 | xargs -0 sha256sum > empty_manifest.txt
echo "without -r: exit $? :: $(cat empty_manifest.txt)"
find none -name '*.mp4' -print0 | xargs -0 -r sha256sum > empty_manifest.txt
echo "with -r:    exit $? :: lines $(wc -l < empty_manifest.txt)"

without -r: exit 0 :: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -
with -r:    exit 0 :: lines 0

Without -r, xargs ran sha256sum once with no filename arguments. sha256sum with no arguments reads standard input, which xargs had connected to an empty source, so it printed the hash of zero bytes, labelled -, and exited 0. The manual covers the behaviour that causes it in the description of -r: "Normally, the command is run once even if there is no input."

A manifest with that line in it looks like a day with one file. Anything that later compares manifests by line count, or tries to verify that entry, gets a misleading answer. With -r the manifest was empty, as intended. In findutils 4.8.0 -r worked together with -0 as documented. I did not test other versions.

A clip that disappears mid-run

The gap between listing the files and hashing them is real. To check what the manifest looks like when a clip is deleted in that gap, this script saves the list first, removes one clip, and then runs xargs on the saved list:

#!/bin/bash
# One clip that vanishes between find and sha256sum.
find clips -name '*.mp4' -print0 | sort -z > list.bin
rm "clips/0004_side_gate.mp4"
xargs -0 sha256sum < list.bin > partial.txt
echo "xargs exit: $?   manifest lines: $(wc -l < partial.txt)"
printf 'd' > "clips/0004_side_gate.mp4"

sha256sum: clips/0004_side_gate.mp4: No such file or directory
xargs exit: 123   manifest lines: 4

Four lines and status 123. Nothing in the manifest itself shows that a fifth file was expected, so the exit status is the record that it happened. A job that pipes the output to a file and moves on without checking the status loses that record.

Skipping the pipe: find -exec with a plus

find can batch the filenames itself with -exec ... {} +. There is no text stream between the two programs, so there is nothing to split or parse. I tested it against the same folder with one clip made unreadable, and against the empty folder:

#!/bin/bash
# Does find report a failed sha256sum in the -exec ... + form?
chmod 000 "clips/0001_garage.mp4"
find clips -name '*.mp4' -exec sha256sum {} + > manifest_e.txt
echo "find exit: $?   manifest lines: $(wc -l < manifest_e.txt)"
chmod 644 "clips/0001_garage.mp4"

sha256sum: clips/0001_garage.mp4: Permission denied
find exit: 1   manifest lines: 4

#!/bin/bash
# The empty day again, with find doing the batching.
rm -rf none && mkdir none
find none -name '*.mp4' -exec sha256sum {} + > empty_e.txt
echo "find exit: $?   manifest lines: $(wc -l < empty_e.txt)"

find exit: 0   manifest lines: 0

With the unreadable clip, find exited 1 and the manifest had four lines, so the failure is visible in the status. With the empty folder, the command did not run at all and the manifest was empty, with no flag needed. The trade-off is order: find emits names in directory order, not sorted, so two manifests of the same folder can differ line by line while holding the same set of hashes. If the manifest needs a stable order, sort it afterwards rather than sorting the names going in.

What I would put in the job

For a manifest of a folder that people can touch:

1. Use find ... -exec sha256sum {} +, or find ... -print0 | sort -z | xargs -0 -r sha256sum if the names need to go in sorted.
2. Check the exit status of the whole pipeline and treat any non-zero value as an incomplete manifest. In bash, set -o pipefail makes a failure in any part of the pipe count.
3. Verify with sha256sum -c rather than with a script that splits lines on spaces, since filenames with spaces and escaped newlines are both legal entries.
4. Treat a manifest whose only line is the empty-input hash ending in - as an empty day, not as one file.

Things I did not test

• Other xargs implementations. Everything ran on GNU findutils 4.8.0. BSD and BusyBox xargs have their own option sets and error behaviour, and I did not try them.
• Very large folders. xargs splits long argument lists into several runs of the command. Each run here fit in one invocation, so I did not see how a failure in one batch interacts with the others.
• Filenames in other encodings. All names here were ASCII.
• Parallel runs with -P. The output order and the exit status rules with several sha256sum processes at once are a separate question.

Sources. Quotes are from the GNU xargs(1) manual page: the description section, the entries for -0 and -r, and the exit status table. Test results come from the scripts above, each run once in a fresh folder on one Linux machine, with the output pasted unedited.

If you want an old phone to record clips in the background so there is a folder to build manifests for, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

A manifest proves only the files that reached the hashing command, so the job has to prove that every file got there.