The error messages a Rust rewrite got subtly wrong: what a black-box probe found in uutils coreutils

When you rewrite a command-line tool, the happy path is the easy part. Output for valid input gets compared early and often. Error messages are where rewrites quietly drift: a different wording, a missing file name, a raw (os error 20) instead of the sentence a script or a person expects.

I have been running behave against uutils coreutils, the Rust rewrite of GNU coreutils. behave probes a CLI as a black box (flags from --help, stdin, files, odd inputs), records what the reference tool does, and replays the same cases against the rewrite. I record from GNU 9.12, the release uutils targets.

One probe case added in v0.1.7 is simply "a regular file passed with a trailing slash" (f/). It turned out to be a good bug finder, because every utility has to turn ENOTDIR into a message, and each one does it a little differently.

What it found

All of these were reported upstream with a fix PR and a test:

• cp f/ dest printed Not a directory (os error 20); GNU says cannot stat 'f/': Not a directory (#15691, #15692)
• shred dir said Not a file; GNU says failed to open for writing: Is a directory (#15630, #15631)
• mv missing/ g reported the wrong reason for the source (#15704, #15705)
• ln f/ link blamed the link instead of the source (#15713, #15714)
• mv a missing/b printed only the OS error, no cannot move 'a' to ... (#15721, #15722)
• mv -T f d left the source out of cannot overwrite directory (#15724, #15725)
• paste open and read errors without the file name (#15642)
• sort -c dir exited 0 with no message at all (#15615, fixed in #15616, merged)

And one that is not about messages: mv -n z z fails with 'z' and 'z' are the same file, while GNU skips the move and exits 0, because it never runs the same-file check when it would not replace the target anyway (#15824, #15825).

Why these slip through

None of these are hard to fix. Most patches are a few lines plus a test. They slip through because a rewrite's own test suite tends to assert what the authors thought of, and nobody thinks of cp f/ dest. A black-box recording of the original tool asserts everything it actually does, including the boring corners.

Two lessons from the review rounds:

1. Match the reference release. GNU 9.7 and 9.12 differ in small ways (quoting in tail -v headers), so recording from an old GNU produces false positives.
2. Exact OS error text differs between platforms. Tests that compare full strerror text need to be Unix-only, while the context part (cannot move 'a' to 'b') can be checked everywhere.

Try it

brew install emirmrkaljevic/tap/behave
# or: go install github.com/emirmrkaljevic/behave/cmd/behave@latest
behave probe cp                                        # record the reference (GNU cp)
behave diff behave-cp --impl ./target/release/cp --skip-tag meta   # replay against your rewrite

If you are porting a CLI to Rust or Go, I would love to hear which cases behave misses.