Conclusion first: a free model drafted a working C++17 directory hasher in one pass. The draft compiled, ran, and was still wrong. A differential test against standard system tools found three real bugs before the tool ever touched a production cache. Generation was the cheap part. Verification was the deliverable.
Background
I needed a deterministic hash of a directory tree. The use case was cache invalidation for a small build pipeline: if any file content, name, or symlink target changes, the cache key must change. If nothing changes, the key must stay identical across machines and across checkouts.
Hand-writing the tool is maybe 200 lines of std::filesystem code. The happy path is easy. The risk lives in ordering, symlinks, and metadata leaking into the hash.
I turned the task into an experiment. MonkeyCode's free model access and free server option meant the model ran on a remote server while I kept verification on my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The plan: let the model write the first version, then prove or disprove it against a reference oracle.
The Contract
The goal was not "a tool that compiles." The goal was a tool that matches a reference implementation on every input I could generate. I wrote the contract in three sentences:
- Same tree → same hash, on any machine.
- Different content, name, or symlink target → different hash.
- File metadata (mtime, inode) must not affect the hash.
Implementation
Step 1: the prompt. I gave the model the contract, the C++17 standard, and one constraint: a single file with no dependencies beyond the standard library.
Step 2: the draft. The model returned one .cpp file in a single response. It compiled on the first try. That is the exact moment where most workflows stop. This one did not.
Step 3: the reference oracle. Instead of reviewing the code line by line, I built a harness that compares the tool against a shell pipeline:
find "$tree" -printf '%P\0' | sort -z | while IFS= read -r -d '' f; do
if [ -L "$tree/$f" ]; then
printf 'L:%s:%s\0' "$f" "$(readlink "$tree/$f")"
elif [ -f "$tree/$f" ]; then
printf 'F:%s:%s\0' "$f" "$(sha256sum "$tree/$f" | cut -d' ' -f1)"
fi
done | sha256sum
The pipeline normalizes a tree into a sorted stream of records, then hashes the stream. It is slow. It is also unambiguous. That is exactly what an oracle should be.
Step 4: the fixture generator. A small script created random trees: nested directories, empty files, duplicate names in different directories, symlinks pointing inside and outside the tree, and files with identical content. I generated 1,000 trees.
Results: Three Bugs
The first differential run failed on 214 of 1,000 trees. The failures clustered into three root causes.
| # | Symptom | Root cause | Fix |
|---|---|---|---|
| 1 | Same tree, different hash on another machine |
recursive_directory_iterator order is unspecified |
Collect paths, sort, then hash |
| 2 | Retargeted symlink produced the same hash | Draft hashed the pointee's content | Hash the link target string |
| 3 | mtime-only change produced a new hash |
last_write_time was part of the hash input |
Hash path, type, content only |
Bug 1 was the subtle one. The tool was deterministic on one machine and wrong everywhere else. The fix was small:
std::vector<std::filesystem::path> paths;
for (auto it = std::filesystem::recursive_directory_iterator(root);
it != std::filesystem::recursive_directory_iterator(); ++it) {
paths.push_back(it->path());
}
std::sort(paths.begin(), paths.end());
After the three fixes, the tool matched the reference on all 1,000 trees. I ran a second batch of 500 trees with deeper nesting and longer paths. Zero mismatches.
Timing: the model's draft took under a minute. The three fixes took about an hour, including the harness. The harness also caught two bugs in my own fixture generator. The oracle does not care who wrote the wrong code.
Lessons Learned
Compiles and runs is the lowest bar. The model passed it instantly and violated the actual contract three times.
A reference oracle turns review into measurement. I did not spot the bugs by reading. The harness pointed at the exact failing input, and the diff told me which part of the contract broke.
Separating generation from verification made the gate hard to skip. The model ran remotely; the oracle ran locally. There was no path from "the model said it works" to "it is merged."
The bugs were not exotic. They were the three sentences I wrote in the contract. The model did not read them carefully enough. Neither would I, on a first pass.
Limitations
Do not copy this workflow blindly. The reference pipeline ignores ACLs, extended attributes, and hard links. My tool ignores them too, by design. If those matter for your tree, write the oracle first and the tool second.
The free model also had a blind spot: it never asked what "deterministic" meant across machines. It assumed local consistency. That assumption was the entire problem.
Closing
The artifact that mattered was not the generated code. It was the 30-line harness that said "wrong" and pointed at the input. If you try this with a free model endpoint, build the oracle before you read the output. The free tier is enough to run the experiment; the gate is what makes it useful.
Top comments (0)