quintetkit

Posted on Sep 5

Treating "It Worked" as Verification Ships Code That Only Works With One Commit

#testing #git #softwaredevelopment #typescript

I shipped a bug in a script I sell. I had also reported that script as
"verified across four scenarios."

I had verified it. In a way that verified nothing.

What was broken

The script cleans up git worktree directories once their branch is merged.
Part of the decision was detecting squash merges.

# the broken check
cherry="$(git cherry "$base" "$branch")"
[ -z "$cherry" ] && return 1
printf '%s\n' "$cherry" | grep -q '^+' && return 1
return 0   # every line is "-", so it is merged

git cherry marks each commit - or + depending on whether an equivalent
exists on the base. All - means merged.

That is wrong. git cherry matches on patch-id, so a squash that collapses
several commits into one can never match.

Measured locally:

# after squash-merging a branch with one commit
$ git cherry main one
- 52b3d961...

# after squash-merging a branch with two commits
$ git cherry main two
+ 02577c73...
+ 221f83ce...

It only works by accident when the branch has a single commit.

Why I did not catch it

I had built a scratch repository and run four scenarios:

no work yet → keep ✓
ordinary merge → delete ✓
unmerged → keep ✓
squash merge → delete ✓

All four behaved as expected. And the branch in the fourth scenario
had exactly one commit.

The test setup read like this:

git checkout -qb squash main
echo x > x.txt && git add -A && git commit -qm c1   # ← one commit
git checkout -q main && git merge -q --squash squash && git commit -qm "squash"

Reaching for the smallest possible reproduction is a normal instinct.
That "smallest" happened to be the only shape in which the broken code passes.

A squash collapses several commits into one. A one-commit squash has none of the
properties of a squash. I thought I was testing squash merges; I was testing
something else that shared the name.

The fix was not "detect it more cleverly"

My first instinct was to find a more accurate check — compare diffs, compare tree
hashes.

I dropped that. A false "merged" verdict deletes someone's working tree.
An error that errs toward deleting costs far more than one that errs toward keeping.

So the answer was to not decide at all:

  # Squash merges are deliberately not detected here.
  #
  # git cherry matches on patch-id, so a squash that collapses several commits
  # into one can never match. Measured: a one-commit branch returns `-`, a
  # two-commit branch returns `+ +`. It only ever worked by accident.
  #
  # A false "merged" verdict deletes a working tree, so we do not guess.
  # With gh available, the PR state answers this exactly.
  return 1

And then made the inability visible:

if ! command -v gh >/dev/null 2>&1; then
  echo "  Note: without gh, squash-merged branches cannot be detected."
  echo "        If any of the kept worktrees were squash-merged, remove them by hand."
fi

Silently doing nothing reads as being broken. If you cannot decide, say so.

Rebuild the scenarios around the property, not the boundary

Branch	Commits	Expected	Result
no work	0	keep	✓
ordinary merge	2	delete	✓
unmerged	2	keep	✓
squash (1 commit)	1	keep	✓
squash (3 commits)	3	keep	✓

The change was not "use more than one commit everywhere." It was splitting
one-commit squash and multi-commit squash into separate cases.

Thinking in boundary values gets you 0, 1, 2. That is not the same as asking what
the smallest shape is in which the property under test actually appears.
A squash's property is collapsing, and collapsing needs at least two things to collapse.

The same hole, in how I measured

Three separate times in the same session I made this mistake:

# wrong. $? belongs to head
./script.sh | head -5
echo "exit code: $?"

Through a pipe, $? is the last command's status. head always succeeds, so
a failing script is reported as exit code 0.

All three times the conclusion was the same: the tool was right, my measurement was wrong.

# right
./script.sh > /tmp/out.log 2>&1
echo "exit code: $?"
head -5 /tmp/out.log

The common thread is never doubting the measurement apparatus.
When a number matches expectations, nothing tests how it was obtained.

The checklist that came out of it

"It worked" is not verification. Can you explain why it passed?
Does the test input actually carry the property under test, or did minimising the repro delete that property?
Decide in advance which way an error should fall. Deleting costs more than keeping.
When you cannot decide, do not go quiet about it.
Distrust the measurement path — pipes, redirection, $?. Especially when the number is the one you wanted.

Summary

Detecting squash merges with git cherry is wrong: it matches on patch-id, so a collapsed squash never matches
It passes for single-commit branches, which is exactly why people try it and ship it
When you minimise a reproduction, check you have not minimised away the property
If you cannot decide accurately, do not decide — but do not stay silent either
Doubt your measurement most when the result is what you expected

This script is something I sell. When a shipped thing turns out to be broken,
writing up how it broke leaves more trust behind than quietly swapping the file.

Related

I publish the configuration for splitting Claude Code into separate personas —
Architect, Coder, Reviewer, Conflict Resolver — under MIT. Copy it, run
./setup.sh, and it works. It does not depend on your tech stack.

https://github.com/quintetkit/quartet

I built one real tool using nothing but this workflow. Every Issue, PR, review
and merge is still there. The parts that went wrong were not deleted.

https://github.com/quintetkit/mdlinkcheck

The version that adds a UI Designer persona, review criteria, a per-Issue
parallel execution script and a 10-chapter guide is on the
product page.

The full kit — five personas, the scripts and the complete guide in English and Japanese — is on BOOTH, a Japanese store with an English interface that takes international cards.

https://quartet-dev.booth.pm/items/8807156

DEV Community