I've been building behave, a tool that probes a command-line program as a black box (its --help, every flag, flag pairs, every value the help lists, odd stdin, bad values), records what it does, and replays those cases against another implementation. The test bed is GNU coreutils vs uutils, the Rust rewrite.
Last week ls and du looked suspiciously good. All 48 generated ls --sort=…, --format=…, --time=… value cases passed against uutils. The reason was boring: every case ran in a fresh, empty temp directory. ls --sort=size on nothing prints nothing, in both implementations.
Give directory tools something to look at
behave now drops a small tree into the work dir when a tool's help text talks about "the current directory" or recursion (ls, du, rg, grep -r):
.hidden README.md a.txt data.csv file10.log file2.log sub/nested.txt
The names, sizes and extensions are picked so that sorting by name, size, version (file2 vs file10), extension and time all give different orders. --sample-files on|off overrides the guess.
Making ls -l and ls -t deterministic
Real files bring real timestamps, and golden tests hate those. Three things made the recordings stable:
-
Fixed mtimes. Every input file gets a fixed modification time, one hour apart in name order, so
ls -thas a real order andls -lprints the same date every run. -
A private parent directory.
ls -alists... If the work dir sits directly in/tmp,..is/tmp, whose link count and mtime change whenever anything else on the machine creates a temp dir. Each case now runs in<tmp>/behave-XXXX/work, and both directories get fixed times. -
Write files in reverse name order. You can't set ctime or birth time. Files created within the same clock tick tie, and
ls -cthen falls back to name order; files that straddle a tick come out newest first. Creating the files in reverse name order makes "newest first" and "name order" the same list, sols -candls --time=birthprint the same output either way.
After that, 500 generated ls cases replay 15 times in a row against GNU ls with zero failures. The 8 cases marked flaky are the --inode ones, which is honest: inode numbers really do change.
What it found
Against uutils 0.12.0, with stderr wording ignored, the interesting new differences were in du's grand-total line:
$ du --total --separate-dirs # GNU
28 .
36 total
$ du --total --separate-dirs # uutils
28 .
28 total
GNU's total still adds up every directory it measured (the sub dir's 8 plus the top's 28), uutils only counts the top level. And du --total --time prints the time column on the total line in GNU but not in uutils. Both turned out to be already listed in uutils' fuzzer tracking issue (#15254) but had no fix yet, so I sent one: #15455, currently in review.
One difference I did not report: ls --hyperlink ends its OSC 8 links with ESC \ in uutils and with BEL in GNU 9.7. Both are valid terminators and uutils chose ESC \ on purpose, so behave's ansi normalizer now strips both forms instead of flagging it.
Earlier the same approach found a sort --check --merge bug in uutils; that fix (#15381) is merged.
Try it
All of this is in v0.1.5.
go install github.com/emirmrkaljevic/behave/cmd/behave@latest
behave probe du --out behave-du
behave diff behave-du --impl /path/to/other/du --ignore-stderr
It's MIT, a single Go binary, and also runs as an MCP server if you want a coding assistant to drive the probing. Issues and ideas welcome on GitHub.
I used AI assistance while writing this post; every number above comes from runs on my machine with GNU coreutils 9.7 and uutils 0.12.0.
Top comments (1)
One more variable worth pinning: locale and timezone. Fixed mtimes only produce stable output if every run also agrees on TZ and LC_TIME, since ls -l formats dates per locale and picks a different year depending on the timezone. Pinning TZ=UTC and LC_ALL=C in the runner removes a whole class of golden diffs that only show up on someone else's machine. The reverse-order creation trick for ctime ties is a neat find too.