The bug that only existed on the platform nobody was looking at

I ship a Python tool with a regression suite of several thousand checks and CI on ubuntu, macOS and
Windows at Python 3.8 and 3.13. Yesterday all five jobs were green. Today those same jobs turned up
four defects that had already shipped.

None of them errored. Each was a rule that was true on the machine it was written on.

1. A guard that compared a resolved path to an unresolved one

The tool has a command that reads one file into the model's context. It refuses a symlink that a
repository chose whose target sits outside that repository — a repo shipping docs/notes.md ->
~/.ssh/id_rsa should not get that file read aloud.

The guard looked like this:

root = find_root()                      # returns a RESOLVED path
named_inside = any(r in Path(os.path.abspath(target)).parents
                   for r in (root, root.resolve()))
if Path(target).exists() and named_inside and not inside(Path(target), root):
    refuse()

os.path.abspath does not resolve anything. It joins the name onto os.getcwd(). On POSIX that is
harmless, because getcwd() returns a canonical path with no symlinks in it — so both sides always
agreed and the guard always fired.

Windows does not do that. GetCurrentDirectory hands back whatever string the process was started
with, short 8.3 components and all. A repository reached through C:\Users\RUNNER~1\... produced
named_inside = False, and the file outside the repository was read exactly as it had been before
the guard was written. The guard was not weaker on Windows. It was absent.

The fix resolves the containing directory — which has no final symlink in it, so it normalises the
name without following the link being judged:

named_dir = Path(os.path.abspath(target)).parent.resolve()
root_dir  = Path(root).resolve()
named_inside = named_dir == root_dir or root_dir in named_dir.parents

I could not reproduce this on macOS. getcwd() canonicalises even if you chdir through a symlink,
so the asymmetry cannot be staged on the platform I develop on. That matters for the rest of this
post.

2. A carriage return no one would ever see

The tool installs an optional pre-commit hook. It rebuilds an index, stages it, then reads back
which files it wrote and refreshes each one:

mytool --written-files | while read -r name; do
  written=$(mytool --write "$name" | sed -n 's/^wrote -> //p')
  [ -n "$written" ] && git add -- "$written"
done

Under git's bundled shell on Windows, Python ends every line with CR LF. read -r keeps the CR. So
name is generic\r, and --write "generic\r" is not a valid choice — argparse refuses it. The
loop wrote nothing and staged nothing, on every commit, for as long as the loop had existed.

The line above it worked the whole time, because it parses no output. So the index looked healthy
while every generated file stood still, and the README went on describing this hook as the thing
that keeps them current.

tr -d '\r' on both readings fixes it. Then I fixed it wrong: the hook body is a Python string, not
a raw one, so writing \r in the source put a real carriage return into the installed hook — the
exact opposite of the intent, and it broke the hook on all five jobs instead of the one it was
written for. The check I had added alongside read the source text, saw two characters, and
reported the escape as present.

The check that caught it was an older one that reads the installed file.

That is the whole lesson of this section: assert on the artifact, not on the source that produces it.

3. The instruction that did nothing when pasted

On a first run that cannot find its commands on your PATH, the tool prints the line to add:

export PATH="/path/to/bin:$PATH"

To every operating system. On Windows that is wrong three times over — export is not a command,
$PATH is not the variable, : is not the separator. The single instruction a new Windows user is
given did nothing when they pasted it.

And the test asserted "export PATH=" in output — on Windows too. The platform where the advice was
useless was the platform confirming it was correct.

4. Not a platform at all: an interpreter

A symlink loop (a -> b -> a) makes Path.resolve() raise. The walk caught that and dropped the
entry in silence — the one exit from that function that recorded nothing, while every other exit
reports what it refused.

Python 3.13 rewrote Path.resolve() onto os.path.realpath(strict=False), which returns a path for
a loop instead of raising. So the same tree was reported honestly on 3.13 and silently truncated on
3.8 through 3.12, under a coverage bar reading 100% over a smaller set than the one it walked.

The check covering it passed on the version I happen to run.

What actually found these

Not cleverness. Two things:

A failing check that prints what it saw. My suite's helper took a name and a boolean. A failing
check said what was expected and nothing about what happened, which is fine when you can re-run it —
and useless when it only fails on a platform you do not have. One round of CI was spent on four FAIL
lines and a guess, and the guess was wrong. So:

def check(name, condition, saw=None):
    if condition:
        PASSED += 1
    else:
        print(f"  FAIL  {name}")
        if saw is not None:                      # only on failure
            for line in str(saw).splitlines()[:6]:
                print(f"        SAW  {line[:200]}")

The very next run printed a commit stat showing the index refreshed and the generated file
untouched. That is section 2, diagnosed in one round instead of three.

Assert the population, not the member. Every one of these is the same shape: a rule applied to
some members of a set and forgotten in the identical ones beside it. So the checks are derived from
source rather than pinned to the line that was wrong. When I fixed the CR in the hook, the check I
wrote was "every place this script reads output produced by a Python program strips CR", derived
from the hook text — not "line 46 has tr -d".

That habit paid immediately somewhere else. GitHub's push protection refused the release over three
test credentials written as literals. The file already had a helper that assembles fake credentials
at runtime, forty lines above, whose docstring says push protection blocked this repository's first
push over exactly these lines. It had been used for some fixtures and not the ones beside them. I
wrote the check over the whole file instead of the three, and it immediately found four more —
GitLab, GitHub, Stripe, Anthropic — already in the published tree, which would refuse a fork's first
push the same way.

The one that was not about platforms at all

After the release went out I swept the published tree for something else: my own machine.

Two defect comments used a real encoded path as their example — my account name and the directory
layout of unrelated work, sitting in a public repository since a release three versions back. Not a
secret in any scanner's sense, which is exactly why nothing stopped them. A published
CODE_OF_CONDUCT.md printed a contact address that a private security advisory link would have
covered without one.

Both are now checks over every tracked file, not fixes to the four lines that were wrong: no shipped
file may quote a real home directory rather than an invented one, and no published document may
print an address outside the reserved example domains a test is supposed to use. The first one
matched its own sanity assertion on the first run, because that line contains a real-looking path;
it builds the needle at runtime now.

Four gates, and the thing that walked past all of them

By the end of the day I had four checks standing between the repository and a leak: no real home
directory in a tracked file, no address in a published document, no credential fixture written as a
literal, and the project's own redactor — seventy-one credential shapes — swept over every tracked
file against a committed baseline. The tree came back clean on all four.

Then I wrote a fifth check that reads a list of terms instead of guessing a shape. Not a pattern: a
plain list of the specific words that must never appear, kept in a file outside every repository,
because a list of things you must not publish is the most concentrated form of the thing you must
not publish.

It failed on the first run. A test fixture in the shipped suite carried a real operational
constraint from my own production work — an imperative sentence naming a real cluster, pasted in
while debugging months earlier. Not a path. Not an address. Not a credential. Nothing was looking
for it, because every check I had written looked for a shape, and this was just a sentence.

That is the argument for the list. Shape-matching catches the classes you have already been burned
by. The next leak is, by definition, the one whose shape you did not think to describe — and the
only thing that catches it is knowing, concretely, what your own secrets are called.

The uncomfortable part

CI ran green on Windows for months while two of these were live. The jobs were not lying; they were
running checks that asked the wrong question, written by someone who could only picture one platform
while writing them. A cross-platform matrix tells you the suite passes everywhere. It does not tell
you the suite is asking the same thing everywhere.

The tool is chamnan — MIT, Python 3.8+, standard library
only, runs entirely locally. python3 tools/verify_release.py from a clean clone re-runs everything
this post quotes and prints what happened on your machine.