comm -23 Said Nine Camera Clips Were Missing From the Backup. Two Were.

I set up a test folder of camera clips and a backup folder with two of them missing, then ran the line a nightly backup check usually rests on, the one that prints what is in one list and not the other: comm -23 phone.txt backup.txt. It reported nine missing clips.

comm did not hide the problem. It printed a warning and exited with status 1. A job that only wants a number throws both away. Below are the test folders, what comm printed, what the GNU manual says about it, a brute-force check of when comm stays quiet, and the version of the job I would run instead.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it records, names or stores anything. The clips are random bytes made on a Linux machine, and every output is pasted from the scripts shown. Versions:

comm (GNU coreutils) 8.32
find (GNU findutils) 4.8.0
GNU bash, version 5.1.16(1)-release (aarch64-unknown-linux-gnu)
Python 3.10.12

Twelve clips, ten backed up

One clip per hour from 06:00 to 17:00 in a phone folder, and a backup folder where the 09:00 and 12:00 clips did not arrive.

#!/bin/bash
# A "phone" folder with twelve hourly clips, and a "backup" folder
# where the 09:00 and 12:00 clips never arrived.
mkdir -p phone backup
for h in 06 07 08 09 10 11 12 13 14 15 16 17; do
  head -c 2048 /dev/urandom > "phone/2026-11-01_${h}00.mp4"
done
for h in 06 07 08 10 11 13 14 15 16 17; do
  cp "phone/2026-11-01_${h}00.mp4" backup/
done
ls phone | wc -l; ls backup | wc -l

12
10

So the right answer is two files: 2026-11-01_0900.mp4 and 2026-11-01_1200.mp4.

The usual comparison

#!/bin/bash
# The usual way: list both sides, ask comm for lines only in the first.
find phone -name '*.mp4' -printf '%f\n' > phone.txt
ls backup > backup.txt
comm -23 phone.txt backup.txt
echo "exit status: $?"

comm: file 1 is not in sorted order
2026-11-01_0800.mp4
2026-11-01_1000.mp4
2026-11-01_1200.mp4
2026-11-01_0700.mp4
2026-11-01_1500.mp4
2026-11-01_1100.mp4
2026-11-01_0900.mp4
2026-11-01_1400.mp4
2026-11-01_0600.mp4
comm: input is not in sorted order
exit status: 1

Nine names, including seven that are sitting in the backup folder. The two clips that really are missing are in there, mixed in with the others, so a person reading the list would not know which to trust.

The cause is the first input. find printed the names in the order the directory handed them back, which on this machine was not by name. ls sorts its output, so the backup list was in order and the phone list was not. comm walks both files once, top to bottom, and decides what is unique by comparing the current line of each. That only works if both files are in the same order.

What the manual says

The coreutils manual puts the requirement first: "Before comm can be used, the input files must be sorted using the collating sequence specified by the LC_COLLATE locale."

It also says what the warning and the status 1 mean: "If an input file is diagnosed as being unsorted, the comm command will exit with a nonzero status (and the output should not be used)."

And it says when the check runs. With neither --check-order nor --nocheck-order on the command line, "wrongly sorted inputs are diagnosed only if an input file is found to contain unpairable lines." The --help text on this machine words the stricter flag as "check that the input is correctly sorted, even if all input lines are pairable".

One more line from the manual matters for a nightly job. In the section on --total, it suggests portable scripts count with wc, and gives comm -23 file1 file2 | wc -l as the example. That example assumes sorted files. Point it at an unsorted one and wc -l counts whatever comm printed, the warning goes to the terminal or the log, and the count looks like any other count.

The job that only wanted a number

#!/bin/bash
# The same comparison inside a nightly job that only wants a number.
missing=$(comm -23 phone.txt backup.txt 2>/dev/null | wc -l)
echo "clips missing from backup: $missing"

clips missing from backup: 9

This is the shape a nightly check tends to take: 2>/dev/null to keep the cron mail quiet, and a count for the log. With the warning discarded, and the pipeline's status coming from wc rather than comm, nothing in the job's output says the nine is unusable. The number is also plausible: nine missing clips out of twelve is a bad night, not an unbelievable one.

When does comm stay quiet?

The manual's wording left me with a question. If the check runs only when there are unpairable lines, can an unsorted input slip through with a wrong answer and exit status 0? I tried every ordering of up to four names on the phone side against every ordering of up to three names on the backup side, and compared comm -23 with the true set difference each time.

# Every ordering of up to four names on the phone side, every subset
# on the backup side (in any order). Count the runs where comm printed
# the wrong answer AND exited 0, and the unsorted runs that exited 0.
import itertools, subprocess
names = "abcd"
runs = wrong = silent_wrong = unsorted_quiet = 0
for n1 in range(1, 5):
    for p in itertools.permutations(names, n1):
        for n2 in range(1, 4):
            for b in itertools.permutations(names, n2):
                open("p.txt", "w").write("\n".join(p) + "\n")
                open("b.txt", "w").write("\n".join(b) + "\n")
                r = subprocess.run(["comm", "-23", "p.txt", "b.txt"],
                                   capture_output=True, text=True,
                                   env={"LC_ALL": "C"})
                want = sorted(x for x in p if x not in b)
                runs += 1
                unsorted = list(p) != sorted(p) or list(b) != sorted(b)
                if unsorted and r.returncode == 0:
                    unsorted_quiet += 1
                if sorted(r.stdout.split()) != want:
                    wrong += 1
                    if r.returncode == 0:
                        silent_wrong += 1
print("runs:", runs)
print("wrong answers:", wrong)
print("wrong answers with exit status 0:", silent_wrong)
print("unsorted input, exit status 0:", unsorted_quiet)

runs: 2560
wrong answers: 1692
wrong answers with exit status 0: 0
unsorted input, exit status 0: 128

In these 2560 runs, every wrong answer came with a nonzero exit status. That fits my reading of the manual: a wrong answer means some line was misjudged as unique, and a line judged unique is an unpairable line, which turns the check on. Unsorted input did pass quietly 128 times, but in each of those the answer happened to come out right anyway.

That is a small test with single-letter names, on one version. It is not proof that the default check catches every case. It is enough to say the warning is worth reading.

The version I would run

#!/bin/bash
# Sort both lists the same way comm will compare them, and stop on error.
export LC_ALL=C
sort phone.txt > phone.sorted
sort backup.txt > backup.sorted
if ! missing=$(comm --check-order -23 phone.sorted backup.sorted); then
  echo "comm failed; not trusting the list" >&2
  exit 1
fi
printf '%s\n' "$missing"
echo "count: $(printf '%s\n' "$missing" | grep -c .)"
echo "== order-free cross-check"
grep -Fxv -f backup.txt phone.txt | sort

2026-11-01_0900.mp4
2026-11-01_1200.mp4
count: 2
== order-free cross-check
2026-11-01_0900.mp4
2026-11-01_1200.mp4

What changed, line by line:

• export LC_ALL=C makes sort and comm compare names by byte value, the same way. The manual ties the order comm expects to LC_COLLATE, so the safe move is to sort and compare under one locale in one script, rather than trust that two programs or two machines agree.
• Both lists go through sort, including the one from ls that already looked sorted. It costs nothing and removes the question.
• --check-order makes the check unconditional, which is the stricter behavior the help text describes.
• The exit status is tested before the output is used. If comm fails, the job says so and stops, instead of reporting a count.
• The count comes from the stored result, so the number and the list are the same thing.

The last block is a cross-check that does not depend on order at all. grep -Fxv -f backup.txt phone.txt prints every line of the phone list that is not an exact, whole-line match for some line in the backup list. It treats every line of the backup list as a pattern, and I only ran it on a dozen names, so I have no numbers for a large archive. Here it gave the same two names from the unsorted files.

What I would check in an existing job

1. Find every comm call, and look at where each input comes from. find, a database query, a file written by an app, or a list someone edited by hand are all inputs that may not be sorted.
2. Look for 2>/dev/null or a pipe into wc -l right after it. Those are the two places the warning and the status get lost.
3. Run it once with --check-order on real data. If it fails, the old output was not safe to use.
4. Keep the sort and the compare under the same locale. Setting LC_ALL=C at the top of the script is the simplest way.
5. Spot-check the count against one known-missing file. Delete one clip from a test backup and confirm the job names it.

What I did not test

• Other comm implementations. This was GNU coreutils 8.32 on Linux. BSD and macOS comm, and BusyBox, may check order differently or not at all.
• Larger names and real collation. The brute-force test used four single letters under LC_ALL=C. Names with spaces, mixed case or non-ASCII characters under another locale were not part of it.
• File names with newlines. A name with a newline in it breaks any one-name-per-line list. The manual's -z option and find -print0 exist for that case; I did not test them here.
• Directory order on other file systems. The phone list came out unsorted on this machine. On another file system find may return names in a different order, sorted or not, which is the reason to sort rather than rely on it.

Sources. Quotes about ordering, the order check and exit status are from the GNU coreutils manual, comm invocation. The --check-order wording is from comm --help on the machine above. Outputs are pasted unedited from the scripts shown.

If you want an old Android phone recording in the background so there is a folder of clips to back up and check, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

Before a backup check trusts a list from comm, sort both inputs the same way and read its exit status, because an unsorted input makes it print a confident wrong answer.