- Book: Decoupled PHP — Clean and Hexagonal Architecture for Applications That Outlive the Framework
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
The agent's last message reads well. "Moved the invoice total calculation out of InvoiceController into an InvoiceTotals use case. No behavior change. All tests pass." You check. The tests do pass.
The tests may have missed one change. The old controller rounded each line's gross amount to cents and then added them up. The new class adds the raw amounts and rounds once at the end. Take three lines of 0.30 net at 19% tax. Each line is 0.357, which rounds to 0.36, so the old code bills 1.08. The new code adds 1.071 and rounds to 1.07. If every invoice in your test fixtures has a single line, both versions agree, and the suite stays green while the bill drops by a cent.
A green suite proves the refactor didn't break what the tests check. It says nothing about the inputs nobody wrote a test for. GitHub's Scientist, a library "for carefully refactoring critical paths," says in its README: "Tests can help guide your refactoring, but you really want to compare the current and refactored behaviors under load."
You can do a small, local version of that today. Check out the starting point in one worktree and the agent's branch in another, replay the same recorded inputs through both, mask the parts that change on every run, and diff what comes out. The script below is about 50 lines of bash, works on most CLI or HTTP services, and needs only git, jq and curl.
Step 1: put the starting point in its own worktree
A git worktree is a second working folder linked to the same repository. You want one for the agent's branch (you may already have it) and one for the commit the agent started from.
Compare against that starting commit, not against today's main. If main moved while the agent worked, a diff against it mixes the agent's changes with everyone else's. git merge-base finds the commit the branch split from:
cd ~/code/app
base=$(git merge-base main hermes/extract-invoice-totals)
git worktree add --detach ../app-base "$base"
cp .env ../app-base/.env
(cd ../app-base && composer install -q)
Passing a commit hash instead of a branch name keeps you clear of git's rule that a branch can be checked out in only one worktree at a time: the docs say add refuses when the commit-ish "is a branch name and is already checked out by another worktree." A fresh worktree holds tracked files only, so ignored ones like .env and vendor/ have to be copied or installed, which is what the last two lines do.
Same inputs, two versions of the code, one diff.
When you are done, git worktree remove ../app-base cleans it up. Per the docs, it refuses to delete a worktree with modified or untracked files unless you add --force.
Step 2: record inputs that look like production
The cases file is one shell command per line. Each line runs once in each worktree, with APP_URL set to that side's server for HTTP cases. Lines starting with # are skipped.
# tests/diff/cases.txt
php artisan invoice:show 1001 --json
php artisan invoice:show 1002 --json
php artisan invoice:show 1003 --json --currency=EUR
php artisan invoice:show 999999 --json
curl -s -w "\n%{response_code}\n" "$APP_URL/api/invoices/1001"
curl -s -w "\n%{response_code}\n" "$APP_URL/api/invoices/abc"
The invoice:show command stands in for whatever Artisan command or CLI your app exposes. Include the boring cases, the multi-line ones, the missing ID and the malformed one. curl's -w writes out text and information after the transfer, so %{response_code} puts the status code into the output, and a 200 that became a 500 shows up in the diff.
You don't have to invent HTTP inputs. nginx's default combined log format puts the method and path in fields 6 and 7, so real GET paths come straight out of an access log:
awk '$6 == "\"GET" { print $7 }' access.log \
| sort -u | head -200 \
| sed 's|.*|curl -s -w "\\n%{response_code}\\n" "$APP_URL&"|' \
>> tests/diff/cases.txt
Read the result before you commit it. Query strings can carry tokens and personal data, and a path with a $ in it will be expanded by the shell. Keep to GETs unless each side has its own copy of the database: a replayed POST changes the rows the next case reads.
Step 3: the script
Save this as scripts/difftest.sh, chmod +x it, and commit it to main together with the cases file, before the agent starts.
#!/usr/bin/env bash
# usage: difftest.sh BASE_DIR CAND_DIR [CASES_FILE]
set -euo pipefail
base=$1 cand=$2 cases=${3:-$1/tests/diff/cases.txt}
out=$(mktemp -d)
trap 'rm -rf "$out"' EXIT
DAY='[0-9]{4}-[0-9]{2}-[0-9]{2}'
CLOCK='[0-9]{2}:[0-9]{2}(:[0-9]{2})?(\.[0-9]+)?'
ZONE='(Z|[+-][0-9]{2}:?[0-9]{2})?'
TS="${DAY}[T ]${CLOCK}${ZONE}"
UUID='[0-9a-fA-F]{8}-([0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}'
if grep -q '^[^#].*APP_URL' "$cases" \
&& [ -z "${BASE_URL:-}" -o -z "${CAND_URL:-}" ]; then
echo "HTTP cases need BASE_URL and CAND_URL" >&2
exit 2
fi
normalize() {
if jq -e . "$1" >/dev/null 2>&1; then
jq -S "${JQ_FILTER:-.}" "$1"
else
cat "$1"
fi | sed -E -e "s/$TS/<TIME>/g" -e "s/$UUID/<UUID>/g"
}
run_case() { # folder, app url, command, output file
(cd "$1" && APP_URL=$2 bash -c "$3") </dev/null >"$4" 2>&1 \
|| echo "exit=$?" >>"$4"
}
n=0 failed=0
while IFS= read -r line; do
case $line in ''|'#'*) continue ;; esac
n=$((n + 1))
run_case "$base" "${BASE_URL:-}" "$line" "$out/$n.base"
run_case "$cand" "${CAND_URL:-}" "$line" "$out/$n.cand"
if [[ $line == *APP_URL* ]] \
&& grep -q '^exit=' "$out/$n.base" "$out/$n.cand"; then
echo "NO RESPONSE in case $n: $line"
failed=$((failed + 1))
continue
fi
if ! diff -u <(normalize "$out/$n.base") \
<(normalize "$out/$n.cand") >"$out/$n.diff"; then
echo "DIFF in case $n: $line"
cat "$out/$n.diff"
failed=$((failed + 1))
fi
done <"$cases"
echo "$n cases, $failed with differences"
[ "$failed" -eq 0 ]
stdout and stderr go into the same file, and a non-zero exit is appended as exit=N, so a new warning or a changed exit code counts as a difference.
Two guards keep the check from passing on nothing. If the cases file has HTTP lines and either URL is unset, the script stops before it runs anything. And if curl exits non-zero on an HTTP case, for example exit 7, "Failed to connect to host", the case fails even when both sides failed the same way, because two identical errors look like agreement to diff.
If the output parses as JSON, jq -S sorts the keys, because a refactor that builds the array in a different order hasn't changed what a JSON client sees (-S outputs "the fields of each object with the keys in sorted order"). jq parses its input "as a sequence of whitespace-separated JSON values," so a JSON body followed by curl's status code still takes this path. Plain text, an empty file or an exit=N line falls through to cat. Drop -S if a client of yours does depend on key order.
Then sed masks ISO timestamps and UUIDs. Anything else that changes on every run goes in JQ_FILTER, using jq's walk and del:
export JQ_FILTER='walk(if type == "object"
then del(.request_id, .generated_at) else . end)'
For HTTP cases, start a server in each worktree on its own port, for example php artisan serve --port=8001 in app-base and --port=8002 in the agent's worktree, then pass the two addresses as BASE_URL and CAND_URL.
Before you trust a diff between two versions, run the old version against itself:
~/code/app-base/scripts/difftest.sh ~/code/app-base ~/code/app-base
For HTTP cases in this run, point both URLs at the base server. Every difference in that run is noise, so normalize it away until the run is clean. Diffy does the same thing with three instances: it sends each request to a candidate, a primary and a secondary running the last known-good code, and when primary and secondary disagree, "Diffy treats that signal as noise."
Step 4: make the diff the check that ends the task
This is where Hermes IDE fits, and I build it, so weigh this with that in mind. Hermes starts each task (⌘N, or Ctrl+Shift+N on Windows and Linux) on its own hermes/<name> branch in its own worktree, with the task as the agent's first prompt. The agent can be Claude Code, Codex or another CLI agent. That is the second worktree from Step 1, already made.
The ⌘N New task sheet: the task, chips for Claude Code, the orders-api project, a new hermes/ worktree, Accept edits, model and effort, and a preview of the claude command Hermes will run.
Hermes reads a .hermes/worktree.toml recipe from your repo. Its done_when list holds the default checks for a task, and as the README puts it, "your own checks can decide when a task is done." copy only takes files git ignores, so .env has to be one.
# .hermes/worktree.toml
setup = ["composer install -q"]
copy = [".env"]
done_when = [
"composer test",
"""BASE_URL=http://127.0.0.1:8001 \
CAND_URL=http://127.0.0.1:8002 \
~/code/app-base/scripts/difftest.sh ~/code/app-base .""",
]
The triple-quoted string is TOML's multi-line form: a backslash at the end of a line trims the newline and the indentation after it, so the second check is one command.
Create the base worktree before you start the task, or the first check has no script to run. At that point the tip of the branch the task starts from is the commit the agent will branch from, so git worktree add --detach ../app-base main gives you the same folder Step 1 does.
With no third argument, the script reads the cases from the base folder. So both the script and the cases come from the base worktree, which sits outside the agent's worktree. An agent told "make this check pass" could otherwise edit cases.txt until the diff goes quiet. The . assumes the check runs from the task's worktree, the same place composer test has to run. If yours starts somewhere else, pass the worktree's path instead. The Review Desk's Worktrees tab lists the task worktrees with their branches.
The Review Desk's Worktrees tab: one Hermes worktree each for orders-api, web-app and orders-types under ~/code, with each worktree's hermes/ branch, task, age and disk size, and the free disk space.
The HTTP cases need both servers running on the ports in those two URLs. Hermes opens plain shells as well as agents, and split panes put them side by side: the base server in one, the branch's server in the other. When the check passes, the Review Desk (⌘G, or Ctrl+Shift+G) shows the change by turn, and Land commits, opens a pull request or merges into the branch the task started from, with undo.
Only put the diff in done_when for tasks that promise no behavior change. On a feature task the diff is supposed to be non-empty, so a check that fails on any difference would block the very change you asked for. For those, run the script by hand and read the diff as a review aid.
Where this breaks
The diff covers the inputs you recorded and nothing more. If the cases file has no multi-line invoice, the rounding change from the top of this post still gets through. Spend your effort on the cases file, and add one every time a bug reaches production.
Anything with side effects needs isolation. Emails, queue jobs, payments and writes to a shared database run twice, once per side. Point both worktrees at fakes or at separate copies, or leave those paths to the test suite.
Non-determinism you can't mask will keep the check red. Unordered lists from the database can be sorted in JQ_FILTER. Random sampling needs a fixed seed. If a result depends on the wall clock, drop the case. And the timestamp regex is coarse: it masks every date-time in the output, including one the refactor might have broken. If dates matter to the code under change, mask by field name in JQ_FILTER instead.
If this was useful
The refactor at the top, logic moving out of a Laravel controller into a use case behind a port, is the move Decoupled PHP is about: getting your domain out from under the framework one slice at a time. A differential check like this one is how you prove each slice moved without changing what it does, whether you made the move or an agent did. If you want to try the Hermes side, it is free and source-available (BSL 1.1) on macOS, Windows and Linux: download it or read the code on GitHub.
Available on Kindle, Paperback, and Hardcover. English, German, and Japanese editions out now — Portuguese and Spanish coming soon.
Top comments (0)