DEV Community

Super Funicular
Super Funicular

Posted on

Six Camera Clips, Three in the Archive, Every mv Exited 0: Testing Same-Name Moves From Two Folders

#bash #linux #shell #debugging

Two phones recording into two folders will sooner or later produce two clips with the same name. I set that up on purpose, with three clips in each folder named the same way, and moved everything into one archive folder four ordinary ways. Three of them ended with three clips in the archive instead of six, and every mv in those three exited 0.

The one form that refused to lose anything was the one that looks the least careful: a single mv with both folders on one command line. Below are the test scripts, what each one printed, what the GNU coreutils manual says about -n, -i and --backup, where this old mv and the current manual disagree about exit status, and the version of the move I would run instead.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it names or stores clips. The clips here are one-line text files made on a Linux machine, and every output is pasted from the scripts shown. Versions:

GNU bash, version 5.1.16(1)-release (aarch64-unknown-linux-gnu)
mv (GNU coreutils) 8.32
PRETTY_NAME="Ubuntu 22.04.5 LTS"
Enter fullscreen mode Exit fullscreen mode

The setup

Each test starts from the same two folders. The first line of every clip says which camera recorded it, so the archive can be checked by reading it rather than by trusting a count.

# Two cameras, same clip names, one archive folder.
# Each clip's first line says which camera recorded it.
setup() {
  rm -rf cam1 cam2 archive
  mkdir cam1 cam2 archive
  for n in 0001 0002 0003; do
    printf 'cam1 clip %s\n' "$n" > "cam1/clip_$n.mp4"
    printf 'cam2 clip %s\n' "$n" > "cam2/clip_$n.mp4"
  done
}
report() {
  echo "archive: $(ls archive | wc -l) files," \
       "left in cam1: $(ls cam1 | wc -l), cam2: $(ls cam2 | wc -l)"
  head -q -n 1 archive/*
}
Enter fullscreen mode Exit fullscreen mode

One mv, both folders 

. ./setup.sh; setup
mv cam1/*.mp4 cam2/*.mp4 archive/
echo "exit $?"
report
Enter fullscreen mode Exit fullscreen mode 
mv: will not overwrite just-created 'archive/clip_0001.mp4' with 'cam2/clip_0001.mp4'
mv: will not overwrite just-created 'archive/clip_0002.mp4' with 'cam2/clip_0002.mp4'
mv: will not overwrite just-created 'archive/clip_0003.mp4' with 'cam2/clip_0003.mp4'
exit 1
archive: 3 files, left in cam1: 0, cam2: 3
cam1 clip 0001
cam1 clip 0002
cam1 clip 0003
Enter fullscreen mode Exit fullscreen mode

GNU mv noticed that the second clip_0001.mp4 would land on a file it had created moments earlier in the same command, refused, said so three times and exited 1. Nothing was lost: the cam2 clips stayed where they were. As far as I can tell from the outputs, this check only covers files the same mv command just moved there. The next two tests are what happens without it.

One mv per camera 

. ./setup.sh; setup
mv cam1/*.mp4 archive/; echo "cam1 exit $?"
mv cam2/*.mp4 archive/; echo "cam2 exit $?"
report
Enter fullscreen mode Exit fullscreen mode 
cam1 exit 0
cam2 exit 0
archive: 3 files, left in cam1: 0, cam2: 0
cam2 clip 0001
cam2 clip 0002
cam2 clip 0003
Enter fullscreen mode Exit fullscreen mode

Same clips, same destination, split into two commands. Both exited 0. The archive holds three files, both camera folders are empty, and every surviving clip is from cam2. The cam1 recordings were replaced, with no message.

A loop, one mv per clip 

. ./setup.sh; setup
for f in cam1/*.mp4 cam2/*.mp4; do
  mv "$f" archive/
done
echo "loop exit $?"
report
Enter fullscreen mode Exit fullscreen mode 
loop exit 0
archive: 3 files, left in cam1: 0, cam2: 0
cam2 clip 0001
cam2 clip 0002
cam2 clip 0003
Enter fullscreen mode Exit fullscreen mode

This is the shape most scripts take, because a loop is where you add logging, skip partial files or build names. Same result as the previous test: three clips survive, all from cam2, and the loop exits 0. Nothing in the output says three recordings are gone.

What the manual says

The GNU coreutils manual's page for mv describes the three options a script would reach for here.

On -n: "Do not overwrite an existing file; silently fail instead."

On -i: "Prompt whether to overwrite each existing destination file, regardless of its permissions, and fail if the response is not affirmative."

On --backup: "Make a backup of each file that would otherwise be overwritten or removed."

And at the bottom: "An exit status of zero indicates success, and a nonzero value indicates failure."

The page I read is the current online manual. The mv I tested is coreutils 8.32, the one Ubuntu 22.04 ships, and the next two tests are where those two differ.

mv -n 

. ./setup.sh; setup
mv -n cam1/*.mp4 archive/; echo "cam1 exit $?"
mv -n cam2/*.mp4 archive/; echo "cam2 exit $?"
report
Enter fullscreen mode Exit fullscreen mode 
cam1 exit 0
cam2 exit 0
archive: 3 files, left in cam1: 0, cam2: 3
cam1 clip 0001
cam1 clip 0002
cam1 clip 0003
Enter fullscreen mode Exit fullscreen mode

-n protected the archive: the three cam1 clips are intact. It also left the three cam2 clips sitting in their folder and exited 0 both times. On this version it is silent, as described, but the exit status does not report a failure. A script that checks $? after mv -n and then empties or deletes the source folder would be deleting the cam2 clips, which exist nowhere else.

The current manual points to --update=none as the replacement for -n, which "will skip existing files but not fail", and describes a none-fail value where "any skipped files are diagnosed and induce a failure." This mv does not have either:

. ./setup.sh; setup
mv cam1/*.mp4 archive/
mv --update=none cam2/*.mp4 archive/; echo "cam2 exit $?"
Enter fullscreen mode Exit fullscreen mode 
mv: option '--update' doesn't allow an argument
Try 'mv --help' for more information.
cam2 exit 1
Enter fullscreen mode Exit fullscreen mode

So a script written against the current manual will stop with a usage error on an older system, and a script written for the older system gets a different exit status from -n on a newer one. If you rely on the exit status of -n, check which version is on the machine that runs the job.

mv -i with nobody at the keyboard

A nightly job has no terminal. Standard input here comes from /dev/null, standing in for a job with no one to answer.

. ./setup.sh; setup
mv cam1/*.mp4 archive/
mv -i cam2/*.mp4 archive/ < /dev/null; rc=$?; echo
echo "cam2 exit $rc"
report
Enter fullscreen mode Exit fullscreen mode 
mv: overwrite 'archive/clip_0001.mp4'? mv: overwrite 'archive/clip_0002.mp4'? mv: overwrite 'archive/clip_0003.mp4'? 
cam2 exit 0
archive: 3 files, left in cam1: 0, cam2: 3
cam1 clip 0001
cam1 clip 0002
cam1 clip 0003
Enter fullscreen mode Exit fullscreen mode

mv printed three prompts, read no answer, skipped all three and exited 0. The archive is safe and the cam2 clips are left behind, same as -n. The manual says -i should fail when the answer is not yes; on 8.32 the skip happened and the failure did not.

mv --backup=numbered 

. ./setup.sh; setup
mv --backup=numbered cam1/*.mp4 archive/; echo "cam1 exit $?"
mv --backup=numbered cam2/*.mp4 archive/; echo "cam2 exit $?"
report
ls archive
Enter fullscreen mode Exit fullscreen mode 
cam1 exit 0
cam2 exit 0
archive: 6 files, left in cam1: 0, cam2: 0
cam2 clip 0001
cam1 clip 0001
cam2 clip 0002
cam1 clip 0002
cam2 clip 0003
cam1 clip 0003
clip_0001.mp4
clip_0001.mp4.~1~
clip_0002.mp4
clip_0002.mp4.~1~
clip_0003.mp4
clip_0003.mp4.~1~
Enter fullscreen mode Exit fullscreen mode

All six clips are in the archive. The cam1 clips were renamed to clip_0001.mp4.~1~ and so on, and the cam2 clips took the plain names. Nothing was lost, and both commands exited 0, which is correct here because nothing failed. The cost is that anything downstream that looks for *.mp4 will not see the backups, so half the footage is there but invisible to a job that globs on the extension.

What I would run instead

The clips collide because the camera folder is part of the identity and the move throws it away. Keeping it in the name removes the collision instead of choosing a winner:

. ./setup.sh; setup
for f in cam1/*.mp4 cam2/*.mp4; do
  cam=${f%%/*}            # cam1
  name=${f##*/}           # clip_0001.mp4
  mv -n "$f" "archive/${cam}_$name"
done
report
ls archive
Enter fullscreen mode Exit fullscreen mode 
archive: 6 files, left in cam1: 0, cam2: 0
cam1 clip 0001
cam1 clip 0002
cam1 clip 0003
cam2 clip 0001
cam2 clip 0002
cam2 clip 0003
cam1_clip_0001.mp4
cam1_clip_0002.mp4
cam1_clip_0003.mp4
cam2_clip_0001.mp4
cam2_clip_0002.mp4
cam2_clip_0003.mp4
Enter fullscreen mode Exit fullscreen mode

Six files, every first line matches its name, both folders empty. I kept -n in the loop as a second line of defence, in case a rerun meets a clip that is already in the archive.

The other thing I would add to any move like this, whatever the naming, is a count that has to balance:

. ./setup.sh; setup
before=$(ls cam1/*.mp4 cam2/*.mp4 | wc -l)
for f in cam1/*.mp4 cam2/*.mp4; do
  mv "$f" archive/
done
after=$(ls archive | wc -l)
echo "before $before, after $after"
if [ "$before" -ne "$after" ]; then
  echo "MISMATCH: $((before - after)) clips missing" >&2
  exit 1
fi
Enter fullscreen mode Exit fullscreen mode 
bash 09_count.sh; echo "script exit $?"
Enter fullscreen mode Exit fullscreen mode 
before 6, after 3
MISMATCH: 3 clips missing
script exit 1
Enter fullscreen mode Exit fullscreen mode

That is the plain loop from earlier, unchanged, with a check around it. It does not save the clips; it turns three missing recordings into a non-zero exit and a line on standard error, which is the part the earlier tests did not produce. A count only proves the numbers match, not that the right files arrived, so on footage I care about I would also compare names, or checksums, before removing anything from the source.

What I would check in an existing job

  1. Find every move or copy into a shared folder and ask whether two sources can produce the same file name. Two cameras, two days with a counter that resets, or a phone reset that restarts numbering will all do it.
  2. Run the job once on purpose against two folders whose clips share names. Read the first line or a checksum of what survives, not the count.
  3. If the script uses mv -n or mv -i, do not treat exit 0 as proof that everything moved. On coreutils 8.32 both returned 0 with files left behind.
  4. Check the version on the machine that runs the job before relying on --update=none or the exit status of -n.
  5. Make an imbalance fail loudly, with a count or a manifest, before anything deletes or clears the source.

What I did not test

  • Newer coreutils. I only ran 8.32. I have not run the current release, so I have not confirmed that its -n or -i exits non-zero where 8.32 exited 0; the manual says they fail, and I am reporting the manual, not a test.
  • Moves across file systems. Every test here moved within one disk, where mv renames. Across disks it copies and then deletes, and I did not test whether anything above changes.
  • BSD and BusyBox mv. macOS and many small Linux images ship a different mv, and the just-created check and the option names may differ.
  • cp. It shares most of these options, but none of the outputs above are from cp.

Sources. Quotes about -n, -i, --backup, --update and exit status are from the GNU coreutils manual, mv: Move (rename) files. Outputs are pasted unedited from the scripts shown.

If you want an old Android phone recording in the background so there is a folder of clips for a job like this to move, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

Before a job moves clips from more than one folder into one place, give it two folders with matching file names and read what survives, because a per-folder or per-file mv replaces the earlier clip and still exits 0.

Top comments (0)