I diffed GNU coreutils against its Rust rewrite with auto-generated tests

#cli #testing #rust #go

"Rewrite it in Rust" is easy to say. The hard part is proving the rewrite does what the old
tool did, including on input nobody thought about: an empty stdin, a file without a trailing
newline, --output-delimiter -1, a directory where a file should be.

So I built behave, a small Go tool that treats
a command-line program as a black box:

probe: read --help and the man page, find the flags, and generate cases: each flag alone, flag pairs, value edge cases (0, -1, abc, empty, missing), stdin samples (empty, unicode, CRLF, 5,000-character line, no trailing newline), the same samples as file arguments, unknown flags and missing files.
record: run each case three times in a fresh temp directory with a fixed environment. If the runs disagree, the case is marked flaky (hello sort --random-sort).
diff: replay every case against another binary and show what changed, per case and per flag.

behave probe sort                       # ~290 cases into ./behave-sort/
behave diff behave-sort --impl ~/uu/sort --skip-tag meta --ignore-stderr

What it found: GNU coreutils vs uutils 0.12.0

tool	matching cases	differences
wc	63/63	none
head	51/51	none
uniq	100/100	none
tr	63/63	none
cut	172/175	`--output-delimiter -1` is rejected as if `-1` were a flag
base64	31/36	`-d` on invalid input: GNU writes the bytes it decoded before failing, uutils writes nothing
sort	~260/290	usage errors exit with 1 instead of 2; `--debug` underline width; `--check --merge`

That's a strong result for uutils: four tools fully match on everything generated. The rest are
small and some are likely deliberate, but you would want to know about them before swapping
binaries in a script that checks $? -eq 2.

jq vs gojq

gojq is a well-known Go implementation of jq that documents its differences. behave finds them
quickly: 331 of 596 cases match (ignoring error wording). The per-flag table shows the reasons:
gojq drops --seq, --ascii-output, --sort-keys (it always sorts keys) and a few others, and
formats huge numbers differently (1e1000 vs 1E+1000).

Making output comparable

Raw byte comparison doesn't work across implementations, so behave normalizes before comparing:

the program name (jq: error vs gojq: error)
temp directory paths, timestamps, ANSI colors, hex addresses
your own regex rules in behave.yaml

Specs are plain YAML, one file per case, so you can commit them next to your port and review
changes in PRs. behave run / behave diff exit non-zero on failures, so they work in CI.

For porting with a coding assistant

behave mcp exposes behave_exec, behave_probe, behave_add_case, behave_list_cases and
behave_diff over MCP, so an assistant porting a tool can check its work after each change
instead of guessing.

Limits

The flag parser is heuristic (tested so far on GNU coreutils and jq help output),
interactive/TTY programs aren't supported yet, and tools that need a meaningful positional
argument (a jq filter, a sed script) need a few samples via --arg.

It's MIT and a single binary for Linux and macOS:
https://github.com/emirmrkaljevic/behave. If you're porting a CLI, I'd like to hear what it
gets wrong.

More of my projects: github.com/emirmrkaljevic

DEV Community