DEV Community: Seth Wheeler

A Curve Fitter That Refuses to Answer

Seth Wheeler — Sat, 26 Sep 2026 03:00:00 +0000

Code: Megapixel99/undetermined

Plenty of libraries fit a curve to measurements and hand you back a number. undetermined hands back a number with the error bar it was decided by, plus an explicit undetermined list carrying a reason on each entry. It will put an observable on that list rather than fit a plateau to a drift. The README's demo adapter flips seeded coins at a ladder of input sizes (8, 32, 128, 512; a rung is one size). heads counts heads among truth flips, so the fair coin's factor of 2 is what settles: it comes back as 1.9978 +/- 0.0032, with the rung its plateau started from. flat ignores truth entirely, and comes back UNDETERMINED, with the reason that no run of 3 rungs agrees and the constant is still moving at the top of the ladder. The second line is the point, and it is also the falsifiable part. A committed pair of tests asserts that the drifting observable lands on the list while the settling one in the same run does not. A tool that always produced the first line would be useless and would still pass every test that checks it produces one.

The scenario is any constant you believe a program has (bytes per record, operations per element, a coin's factor of 2) that has to be measured rather than read off. You supply an adapter that can run the thing at a controllable input size; the library never knows what program it is looking at:

from undetermined import characterize

class Coin:
    def truths(self):
        return [8, 32, 128, 512]                 # the input sizes: a ladder

    @property
    def observables(self):
        return {"heads": heads, "flat": flat}    # each is (truth, seed) -> number

report = characterize(Coin(), trials=2500)
report["undetermined"]                           # ['flat'], with the reason in notes

pip install undetermined and npm install undetermined are the same tree at the same version. It refuses three ways of being confidently wrong. An observable that ignores its seed raises immediately, because the fitter averages over different seeds, and a mean over noise still has a standard error, still forms a ladder, and can still plateau. Every guard downstream compares against that error, so a broken adapter does not produce a wrong-looking answer. It produces a confident one. A constant that never settles is reported with its reason, since a plateau requires three consecutive rungs agreeing within two combined standard errors. And a choice between observables that is not earned is refused too. An observable must vary at least three times its own measurement error across instances to be called informative, and must beat the runner-up by the same factor to be chosen. Even the degenerate case is a raise rather than a shrug. An adapter with one observable leaves the library no choice to be shown making.

The rule underneath all of it is to compare against the noise, never against the size. A spread only means something in units of the error on the thing that spread, and dividing by the magnitude instead is how a large number gets mistaken for a real one.

That rule is also where the library was wrong, and the 0.2.0 release notes the failure in a way I want to quote at length because of one clause. Every error bar used to be Type A, the scatter of repeated draws, sd/sqrt(N). An observable that answers the same number every time has no scatter, so its standard error was zero, the ladder-builder dropped the rung, and the report said the constant could not be determined. It said that about a quantity it had measured exactly, in the same words it uses for a quantity with no constant at all. Everything the library had been measured on was a stochastic simulation, so the case never came up. Pointed at real systems it is the common case. And the clause that stings: a test in the repository asserted the old behaviour. The defect was not merely unnoticed, it was pinned. It surfaced because countfn, a sibling package built on this one, hit it for every deterministic operation count. That package wrote the finding into its README rather than working around it quietly.

The fix is the metrologist's combined standard uncertainty, u = sqrt(u_A² + u_B²), where u_B is granule/sqrt(12) and the granule is the resolution the observations are reported at. u_B is deliberately not divided by sqrt(N). Repeating a deterministic measurement does not buy resolution, and an error bar that shrank when you looped would let any constant be made significant by asking twice. The granule derivation errs fine on purpose. A coarse granule widens every error bar, and wide error bars are how a search flattens a drift that was never constant; the alternatives (a GCD of the values, the spacing between them) can only err coarse, so neither is used. The fixtures ship the control for the fix itself. exact and drifting are both deterministic, exact is now recovered, and drifting is still UNDETERMINED in both halves. A fix that widened error bars until the first one worked would have flattened the second.

Two halves ship from one tree, at one version, on PyPI and npm, and the thresholds are asserted identical down to the explanatory strings. That forced a shared number formatter. %g and toPrecision(6) agree only on integers below 10⁶ and non-integers in [1e-4, 1e6), and %.1f and toFixed(1) round halves in opposite directions. The shared rule renders an exact integer in full (a rung at 16777216 is not clarified by calling it 1.67772e+07) and rounds half away from zero. Both halves are pinned to expected strings independently, since two halves that had drifted together would still agree with each other.

One dependency decision is worth recording because it went the unfashionable way. A nondet edge was considered for the reproducibility precondition and rejected. nondet addresses functions as FILE::NAME, this library's observables are closures with no such address, and wiring it in would have meant a fake path or a check that never ran. A dependency that looks like a guarantee and is not is worse than no dependency, so the precondition is implemented natively in both halves, in the same call that would have needed the guarantee.

A Check With a Zero Denominator Reports Clean

Seth Wheeler — Fri, 25 Sep 2026 03:00:00 +0000

Code: Megapixel99/zerocase

A suite whose every test was skipped, a lint whose glob matched nothing, and a coverage run over no statements can all exit 0. Every one of them has already written the real number down in a file with a field name on it. zerocase wraps a command, reads the machine-readable report the runner wrote, and refuses the run when nothing in the report actually executed:

pip install zerocase        # or: npm install zerocase
zerocase --junit reports/junit.xml -- pytest tests/
zerocase --lcov coverage/lcov.info --min 200 -- npm run coverage

didrun, covered here two days ago, asks the same question of the command's stdout with a regex you supply, and it is this package's one dependency. Everything about states, exit codes and freshness is that package's and is not reimplemented.

What the report parsers add is one distinction, and it is the whole package: a total is not a denominator. <testsuite tests="50" skipped="50"> is a green run of nothing wearing a total, and a stdout pattern counting "50 tests" is satisfied by exactly the run it was meant to catch. So every parser here (JUnit XML, TAP, LCOV, Cobertura, ESLint's JSON) returns two numbers, total and executed, and the floor (the --min count the run must clear, at least 1) applies to executed. A <testcase> carrying both a <skipped/> and a <failure> (a quarantined flake, an xfail, a rerun that gave up) did not run and therefore cannot have failed. Counting it as both is how a suite that skips everything comes to look like a suite that broke everything. The failed count is read and printed and is never a verdict, since whether a suite passed is the exit code's business. A report in which every test failed still satisfies the floor, and there is a test asserting exactly that.

Two more refusals keep the number honest. A report can hold a number, hold a zero, or not be readable at all, and the third must not collapse into either of the others. An unparseable report scored as a zero fails builds for the wrong reason; scored as a pass, it is the defect the tool exists to report, wearing the tool's own badge. And the report must have been written during this run, because yesterday's junit.xml parses beautifully and says four hundred tests passed. --allow-stale lifts that. The name is deliberately unpleasant. Attributes are not trusted either. tests="47" in the header is a claim the writer made, and forty-seven <testcase> elements are the thing itself. When they disagree (a runner killed halfway leaves a header from one run and a body from another), the elements are counted and the discrepancy is printed.

The claim a stranger can check is the pair of committed controls: a report the run wrote with tests in it is evidence, and a report the run wrote with nothing in it is did-not-run. They are asserted together, so a gate that always passes and one that always refuses both fail. The parity suite then sends one table of fixtures to both language halves over stdin and compares the tallies and the sentences character for character. Two halves that each maintained their own inputs would drift by being asked different questions, and would then report agreement about that.

The part that taught me the most was the mutation pass over the source. Thirteen mutations were applied and all thirteen are now caught. Two survived their first run, and both were worth more than the eleven catches. The empty-glob mutation survived because the test covering an empty glob reached its verdict through the freshness gate, which fires first and returns before the second refusal was ever reached. The branch was live, reachable through --allow-stale, and exercised by nothing. The skipped-counted-as-failed mutation survived because no fixture held a <testcase> carrying both a skip and a failure, so the guard was untested by construction. The fixture that now exists is a shape real runners emit. A third mutation was wrong rather than surviving. Mutating self.tally = self.tally or {...} is a no-op on the first run, so it changed nothing and proved nothing, and a mutation that cannot alter behaviour scores as SURVIVED and reads as a test gap. A mutation suite can lie in the flattering direction as well as the other one.

The prior-art sweep is why I am confident the gap is real rather than under-searched; an earlier sweep for a sibling package had to retract four package proposals after checking only keyword variants. On npm, keywords:junit returns 349 packages and every one inspected is a reporter, a merger or a viewer. keywords:test-count returns zero. The two real neighbours are LambdaTest's @testmuai/evidence-cli, which validates the shape of an evidence pack rather than the number in it, and PyPI's evidence-gate, which audits CI evidence bundles after the fact. Nothing found gates a run on the denominator of the report the runner already wrote.

The honest limits: the parsers are scanners rather than XML parsers, with comments and CDATA stripped first, and a deliberately hostile document could still fool them. The committed fixture proves a failure message quoting <testcase> does not invent tests. And it cannot tell a suite that ran from a suite that reported; it moves the claim from the exit code to the report, which is much harder to fake by accident and not impossible.

Breaking CI Guards on Purpose to Prove They Can Fail

Seth Wheeler — Thu, 24 Sep 2026 03:00:00 +0000

Code: Megapixel99/canfail

A CI guard that has never failed may be incapable of failing. A lint rule disabled by a config merge, a type check whose glob stopped matching, a schema validation step pointed at the wrong directory, a security scanner with an empty ruleset. Every one of them is green forever, and green is what you were looking for. For code guarded by tests, mutation tools like Stryker generate the breaks automatically and you should use them. canfail is for the guards they do not cover, where the thing being guarded is YAML, Terraform, a Dockerfile, or anything else without a function to mutate. You declare a break (a file, an anchor string, its replacement, and what failure you expect). The tool applies it, runs your check, and reports whether the check noticed.

pip install canfail
canfail canfail.json

{
  "checks": [{
    "name": "unit tests", "run": ["python3", "example/test_prices.py", "-q"],
    "breaks": [{
      "name": "total: multiply -> add",
      "file": "example/src/prices.py",
      "replace": "item[\"price\"] * item[\"quantity\"]",
      "with": "item[\"price\"] + item[\"quantity\"]",
      "expect": "assert|Error"
    }]
  }]
}

It is a mutation harness, a very small one aimed at CI configuration, so it inherits the rules that make one trustworthy. The check must pass on the clean tree first; a check that was already red tells you nothing when you break something. A failure is not a catch: it has to fail for the reason you named. A break that just makes the file unparseable is scored wrong-failure, and a check that hit its timeout exits 124, which looks exactly like going red and settles nothing. The anchor must match exactly once. Zero matches means the break never happened, and two means the check was asked about code nobody was thinking of. And the file must come back, verified by digest, which is restore-verified's whole job. canfail used to carry 78 lines of that guard inline, about a quarter of the module, so it could claim no dependencies. Deleting the copy once the real package existed was the right trade: a second copy of a guarantee is a second thing to get wrong, and the copy is the one that never gets the upstream's tests.

The example configuration in the repository declares four breaks against one check and yields four different outcomes in a single run, closing with the tally 4 declared break(s): 1 caught, 1 not caught, 2 not settled. CI asserts that tally line rather than just the exit code, because exit 1 alone would be satisfied by finding the wrong thing.

The paragraph I most want to keep is about a fix of mine that measurement killed. The first working version reported a genuinely blind guard as catching, but only when its break ran second. After the first break, Python had written __pycache__ bytecode from the broken source, and the second break's run executed that stale bytecode, failing for the previous break's reason. My fix was to stop restoring mtime on the guarded file. Then canfail got the treatment it gives other people's guards, deliberate breaks to its own source with its suite as the check, and that mutation pass showed the fix does nothing. Re-enabling bytecode caching breaks the test that pins the ordering bug whether or not mtime is restored. mtime invalidation has one-second granularity, this tool edits, runs and restores in milliseconds, and a .pyc written from the broken source therefore looks fresh either way. PYTHONDONTWRITEBYTECODE is the load-bearing guard; not restoring mtime is a cheap belt beside it. The general form is worth carrying to any tool with a fast edit cycle: anything keyed on mtime (bytecode caches, make, ninja, file watchers) is blind on sub-second edits.

blind also means two different things unless you make it say which: the check ran and did not notice, or the check never ran at all. The first says your guard is weak and the second says it is missing. They send you to opposite ends of the CI file. Declaring evidence on a check hands that question to didrun, so a break that stops the check from running becomes a refusal rather than a finding. Omit it and the report says outright that it cannot tell the two apart. An evidence object naming no known predicate is a config error rather than a no-op. A misspelled key silently downgrading the check is this package's own failure mode pointed at itself.

That self-directed suspicion earned its keep. The current README carries a section titled "Things this got wrong about itself", and every entry is a verdict the tool reported without having earned it. A killed check scored as a catch, because exit 124 is non-zero. A wrote evidence path resolved in the tool's directory while the check ran in --cwd, so a check that wrote its report perfectly was reported as never having run. An uncompilable expect regex reached the interpreter as a traceback and exited 1, the code that means a guard is blind. A break applied to a CRLF file came back as a whole-file diff, because universal-newline reading stripped the returns. Each is now pinned by a test that fails without the fix.

The score so far is six mutations against the original properties: five caught, and the sixth (restoring mtime) survived, which is how the paragraph above got corrected. Six more were applied against the later fixes, and all six were caught. The adoption cost is honest too: you have to write the breaks yourself. For Python code guarded by a Python suite, the README points you at PyPI's mutation-testing instead, which swaps __code__ objects and never touches the disk at all. Everything canfail carries about restores and signals is apparatus it needs only because it edits real files. That is what buys it the config formats nothing else covers.

An Exit Code Cannot Say Whether Anything Happened

Seth Wheeler — Wed, 23 Sep 2026 03:00:00 +0000

Code: Megapixel99/didrun

Exit code 0 means "I did not fail". A suite of ten thousand assertions and a suite that collected nothing both report it; no amount of reading the number harder will separate them. That is how a check silently stops checking and nobody finds out for a year. The glob stops matching, the marker deselects everything, the step keeps exiting 0, and green is what everyone was looking for.

The measured example, from didrun's README rather than from folklore: go test ./... on a tree with no test files prints [no test files] and exits 0. Wrapped as didrun --expect "^ok " -- go test ./..., the same run exits 3. The report says nothing in the output matched, there is no evidence this command did anything, and the exit 0 was itself the failure.

The rule the tool is built on is that a check must answer three questions separately: did it run, did it fail, and was the failure the right one. Collapsing any two of those is how every defect of this kind happens. So didrun returns four states rather than a number: ran-and-passed (exit 0), ran-and-failed (the command's own exit code), did-not-run (exit 3), and ran-and-failed-wrongly (exit 4). did-not-run never borrows the command's own code, even when the command failed. "Your tests failed" and "you have no tests" send you to different places. ran-and-failed-wrongly gets exit 4, and exists because "it failed" is not "my check caught something". A suite that dies on a syntax error fails exactly as red as one that caught the defect you planted.

Evidence is what you declare, and declaring none is an error rather than a degradation. With no predicate, the library throws and the CLI exits 2, since a tool that silently falls back to forwarding the exit code is the thing it replaces. The predicate that matters most is --expect-count, and the committed control test is test_zero_passed_is_DID_NOT_RUN_even_though_the_line_matched. A run printing 0 passed in 0.01s with exit 0 satisfies matches(/\d+ passed/) perfectly, because the runner cheerfully prints its zero. The thing that makes a green run meaningless is almost always a zero rather than an absence, so count(/(\d+) passed/) reads the number and scores the run did-not-run. The suite pins both behaviours side by side: the pattern check is fooled, as advertised, and the count check is not.

--wrote PATH is the other predicate with a sharp edge: it is not "the file exists". A junit.xml left over from yesterday exists, and a runner that never started leaves it exactly where it was. So the file must have been created, changed or rewritten during this run, and a byte-identical leftover is reported as the stale thing it is. A timeout is classified rather than swallowed; a killed check did not finish, and must never read as having passed. And the report prints what each predicate looked for and what it found even when everything passed, because a check whose output is only a verdict is one nobody can audit.

Some runners already answer the first question, and the README's table says to let them. pytest exits 5 when it collects nothing, and jest and vitest fail by default when no test matches (their --passWithNoTests flag is the decision this tool exists to argue with). go test, linters handed an empty glob, and every plain shell step in every CI file are the rows this is for. Even the good rows fail the second question, though. A pytest suite in which every test is skipped prints 2 skipped and exits 0 (measured here on pytest 9.1.1), and nothing about that run checked anything.

The package ships as one command from two registries, pip install didrun and npm install -g @megapixel99/didrun. The npm half is scoped because npm refused the bare name as too similar to an existing package called madrun. The release workflow enforces that the two names may differ by the scope and nothing else, and a parity suite asserts the two halves share the four state names, the exit codes, and the same classification of the same run. Six mutations were applied to the source (never reporting did-not-run, treating it as success, requiring only one predicate instead of all, allowing a run with no evidence, ignoring the count floor, accepting a stale artefact) and each was caught by the test that should catch it.

The prior-art sweep found the space emptier than I expected. pytest-custom-exit-code is one runner answering the question for itself, and evidence-gate on PyPI audits GitHub Actions evidence bundles after the fact. ranit intersects a coverage database with a git diff, which is the closest thing in spirit found anywhere. Nothing found wraps an arbitrary command and asks whether it did anything, which is a strange gap for a failure mode this common. Parsing the reports the runner already wrote is a different question, with its own package and its own post to come.

Proving the Tree Came Back After Breaking It on Purpose

Seth Wheeler — Tue, 22 Sep 2026 03:00:00 +0000

Code: Megapixel99/restore-verified

Any tool that breaks a file on purpose and puts it back (a mutation harness, a codemod, a benchmark swapping a config) has four ways to leave the tree wrong. The standard try/finally covers exactly one of them. First, an exception mid-run, which is the covered case. Second, a signal: finally does not run on SIGTERM, so a plain kill leaves the file broken. Third, the restore itself being wrong: a restore that ran is not a restore that worked. And fourth, SIGKILL, which nothing inside the process can cover.

restore-verified is those three uncovered rows as a package, in Python and JavaScript, and using it is one context manager:

from restore_verified import guarded

with guarded("src/parser.py") as g:
    g.write(g.read().replace("<=", "<"))
    run_the_suite()
# restored here, and the restore is checked byte for byte

pip install restore-verified and npm install restore-verified carry the same API and the same CLI, with a callback in place of with on the JavaScript side. The second row is measured rather than asserted. The suite runs the same mutation three ways in a real subprocess and really kills it. The control is test_SIGTERM_leaves_a_try_finally_harness_broken: an ordinary try/finally harness, SIGTERMed mid-run, must leave the mutated file on disk. If try/finally ever survives SIGTERM, the premise of the package is wrong and that test fails saying so. Beside it, the guarded version of the same harness must come back. A third test asserts the signal is re-delivered afterwards, so the process still dies with status -15 and a kill is never converted into "nothing happened".

The third row is where the name comes from. The npm atomic-write packages make a write all-or-nothing, and PyPI's in-place restores on an exception. Nothing I found re-reads what it put back. A restore can run perfectly and still be wrong: a buffer captured after mutating instead of before, a different encoding on the way out, one of the two files you touched. All three leave the restore path looking healthy, and every run after them scores code nobody wrote. So the guard hashes before and compares after, and g.read() hands back the snapshot rather than re-reading the possibly-mutated file. The snapshot lives in a temp directory instead of a foo.py.bak beside the code, because a scratch file in the directory being measured changes what a file walker collects.

The fourth row cannot be fixed and can only be caught. The ordinary way to be SIGKILLed is not an impatient person, it is a timeout. subprocess.run(..., timeout=...) calls kill() when the deadline passes, and so does a CI runner that has waited long enough. A harness carrying a perfect in-process guard, invoked under a timeout it exceeds, leaves the tree exactly as broken as one carrying no guard at all. So the check lives in whatever invoked the harness. A Sentinel records per-file digests into a manifest (one JSON document either language half can read) before the run, verify compares after, and the CLI form is one line: restore-verified run --paths src/ --timeout 600 --restore -- ./harness.sh, which exits 3 when the tree did not come back, its own code, never the command's. Even --restore still exits 3, because a run this thing had to repair is recoverable, not trustworthy. What Mutation Testing Frameworks Do When a Timeout Kills Them is this argument run against four real frameworks. The short version is that frameworks which mutate a copy never need any of this, and the README says so plainly: avoiding in-place mutation is a better answer than guarding it, and the package is for the tools that cannot.

On a clean git checkout you should also just use git, and the README opens with that. git diff --quiet catches an unrestored change for free. The cases that defeat it are asserted in a test class called TheGitControl. On a developer's dirty checkout, a failed restore is indistinguishable from the uncommitted work, and git checkout -- FILE destroys that work rather than restoring it. A snapshot here is per-file and taken when you start; git's is repo-wide and taken at the last commit, and those are the same thing only on a clean tree.

The JavaScript half is not a translation, and its differences were found by failing tests rather than by reading. A Node signal handler cannot unwind an awaited body the way a Python handler raises into a with block. So the JS handler performs the restore itself, synchronously. And a registered signal listener is also a handle keeping the event loop alive. Removing the last one before re-raising the signal can leave the loop empty, so Node exits before the re-raised signal lands; a timer held across the re-raise closes that hole. The parity suite asserts the manifest in both directions: Python restores from a JavaScript manifest, and the two halves produce byte-identical manifests for the same tree.

One design choice cuts against a sibling package and both are right. The guard restores mtime on purpose, so a guarded edit does not trigger a rebuild in anything keyed on file times. canfail opts out of that deliberately, for reasons its README records and a later post here will walk through.

Five mutations were applied to the source, under this package's own guard, and all five were caught: removing the signal installation, removing the verification, dropping the re-delivery, keeping the snapshot beside the code, and making verify always report clean.

A Determinism Check Has to Leave the Process

Seth Wheeler — Mon, 21 Sep 2026 03:00:00 +0000

Code: Megapixel99/nondet

The obvious way to check whether a Python function is deterministic is to call it twice and compare. I wrote that check, and it is blind to the commonest source of nondeterminism in the language. String hashing is randomised per interpreter, so set and dict iteration order is stable within a process and different in every new one. A function returning list({'alpha', 'beta', 'gamma'}) answers identically twenty times out of twenty inside one interpreter. In the transcript the README records, three fresh python3 -c invocations gave three different orderings (six orderings are possible, so a rerun sometimes repeats one, and the arithmetic on that is below).

nondet is the check rebuilt around that fact: point it at a tree or a FILE::NAME, and it re-runs each function against a fixed input ladder in fresh worker processes. The ladder is a deterministic list of probe inputs (a rung is one of them), covering functions of one to three positional arguments; variadics, keyword-only signatures and zero-arity functions are refused with a reason rather than guessed at. PYTHONHASHSEED is cleared for the workers, so a seed fixed in your environment cannot blind the check.

pip install nondet
nondet src/                     # every module-level function under a tree
nondet src/util.py::normalise   # one function

A finding prints the function, the input, and the two answers:

  nondeterministic  src/features.py::resolve_features
         [["alpha","beta","gamma","delta","epsilon"]] -> V:set["epsilon","delta",...]  then  V:set["beta","gamma",...]

The claim worth checking is the control committed as test_in_process_repetition_would_have_missed_it. It asserts that the in-process check finds no variation over 20 calls on a hash-order function, and that nondet finds it anyway. If in-process repetition ever catches it, fresh processes are expensive theatre and the test says so in those words.

The verdicts are deliberately asymmetric. nondeterministic comes with a witness, an input and the two different answers it produced, and a witness is a fact. deterministic means no run disagreed across the ladder, which is the absence of a counterexample rather than proof of one's absence. The output says so instead of letting you read it as a guarantee. The package's own numbers put a magnitude on that gap. A hash-order defect in a three-key dict admits 3! = 6 orderings, so three fresh processes miss it about 2.8% of the time. An eight-key dict admits 40,320 and is missed about once in a billion. The ladder carries both shapes, and the wide one does the detecting.

Scoring it needed labels written down separately from the function names, so the checker is not graded against its own naming convention. On the 19-function fixture set, 9 of 9 nondeterministic functions are caught and 0 of 10 deterministic ones are falsely flagged. One deterministic function comes back as a refusal instead, deliberately. Every rung of its ladder raised, and a ladder that only ever reached a function's type errors has not measured its behaviour. The fixture pairs are the point of the set. dedup_unsorted and dedup_sorted are one sorted() apart. seeded uses random.Random(42): deterministic, and exactly the false positive a static gate that greps for random produces. duration_arithmetic imports time and never reads the clock.

Pointed at a real 283-function tree, it probed 127 with the safety gate on and found 2 genuinely nondeterministic functions. One returns a set, and one's value moves between runs. (That census predates a newer rule that files a function whose every rung raised as a refusal, so a run today probes slightly fewer; the README says so under the table, and neither finding moves.) The gate exists because of the sharpest thing I learned building this: the probe executes the code it is asked about. An early run reported a function raising TypeError once and FileExistsError the next time. That is a true finding, and it is also proof that the probe had just created a file on the disk. The line is drawn at writing and communicating rather than at impurity, since time, random, uuid and set ordering are read-only and are exactly the target. open(), subprocess and sockets are refused. The gate costs recall and the census (the printed count of what was refused and why) says so: --unsafe lifts probing to 169 functions and the findings to 4. One of the two findings it hides (a function returning a path under a fresh temp directory) is a true positive the gated run can no longer see.

The environment is varied between runs too, timezone and locale, an idea taken directly from reprotest. The fixture that justifies it is epoch_year: fromtimestamp(0).year is 1970 in UTC and 1969 west of it, does not move with the clock, and is caught only because the timezone varies.

The part of the README I would keep if I could keep one paragraph is about the tool's own bugs, because both wore the costume of findings about the code under test. Loading a package module by file path broke relative imports and refused 56 of 68 functions; sending the result vector over stdout meant any function that printed corrupted it. Both were caught by pointing the tool at a real codebase and disbelieving the refusal rate. One fix took reach from 2 of 68 functions to 60 of 68 while breaking correctness on all 17 fixtures of the time. A tool watched only by how many functions it probed would have scored that as an improvement.

Prior art was swept on both registries before publishing. On npm, keywords:purity returns 26 packages and every one is static analysis. The real neighbours are reprotest at build granularity, Groce and Holmes (QRS 2020) at test granularity, and the pytest-flakefinder family, which reruns tests. Nothing found points at an arbitrary function, walks a ladder in fresh processes, and hands back the input that distinguished two runs. That is the granularity this fills.

Ten Packages, One Rule: A Check Must Be Able to Fail

Seth Wheeler — Sun, 20 Sep 2026 03:00:00 +0000

Between 20 August and 31 August I published ten small packages, eight of them on the last day. The first of the eight repositories was created at 06:05 UTC and the last at 18:34. They are small verification and measurement tools, each in Python or JavaScript or both, installable from PyPI and npm, and each encodes one way a green check can mean nothing. Nothing about the code was written in a day. The packages are extractions from the measurement work behind much of what is on this site. The recurring failure there was almost never the thing being measured; it was the instrument reporting success without having checked anything. A memory profiler that watched the wrong process for eight months and a rate limiter that reported requests it never sent are both that failure. The research codebase these packages fell out of is, like the ones in those posts, not public. The packages are the public form of what it kept teaching.

The failure has a family resemblance wherever it appears. go test ./... on a tree with no test files prints [no test files] and exits 0. A pytest suite in which every test is skipped prints 2 skipped and exits 0 (measured on pytest 9.1.1). A JUnit report can read tests="50" skipped="50", which is a green run of nothing wearing a total. A lint rule disabled by a config merge is green forever, and green is what everyone was looking for. In each case a check stopped checking, and nothing about the repository looks wrong.

The rule all ten packages are built on is stated in didrun's README: a check must answer separately whether it ran, whether it failed, and whether the failure was the right one. Collapsing any two of those three is how every defect in this family happens. The corollary is the title of this post. A guard that has never failed may be incapable of failing, and the only way to know is to give it something it must fail on.

So here is the claim, and a stranger can check it one repository at a time: every one of the ten commits at least one test that would report the package's own premise wrong. Not a test that the tool works; a test that the naive alternative fails where the tool succeeds, or that fixtures can force the tool's verdict in both directions. If any repository below lacks such a test, this post is wrong about it. Two terms recur, because the family shares its probing machinery: a ladder is a fixed list of probe inputs a tool walks in order, and a witness is an input that produced two different answers. And assay-checks appears twice, because it carries two unrelated questions in one binary and each needs its own control, so the table has eleven rows for ten packages.

package	the way a check lies	the committed control
assay-checks, a duplicate-function finder that executes code	two functions maintained separately answer identically	pairs come from executed outcome vectors, never names, and functions that genuinely differ must not group
nondet, a determinism prober	an in-process repeat calls a function deterministic when hash order only moves between processes	the in-process check must find nothing in 20 calls while fresh processes find the witness
assay's `runners` half, an auditor of mutation harnesses	no failures reported and no test executed look identical; a crash scored as a catch	seven properties, each shipped as a mutation its own runner must catch
restore-verified, a guard for tools that break files on purpose	a restore that ran is read as a restore that worked	a control asserts `try/finally` really does leave the tree broken on SIGTERM
didrun, a command wrapper demanding evidence of work	exit 0 is read as evidence that something happened	`0 passed` matching the expected pattern must still score as did-not-run
canfail, a breaker of guarded files	a CI guard is green because it cannot go red	the example config must yield a catch, a blind guard and two refusals in one run, and CI asserts that tally line
undetermined, a curve fitter that can refuse	a constant is fitted to a drift and reported without the error bar that decided it	the demo's second observable must come back UNDETERMINED while the first does not
zerocase, a reader of the reports runners already write	a zero denominator reports clean	a full report and an empty one from the same command shape must get opposite verdicts
countfn, an operation counter for scaling questions	a complexity class is named because a curve fit looks close	three functions must produce three answers at once: n², log n, and a refusal
ladderpin, a freezer of measured behaviour	behaviour drifts under tests that stay green; a flaky pin blames the pinning tool	with the determinism gate off, a pin on an unchanged tree must go red
lexindex, a code completion engine over repository statistics	a completion accuracy is quoted with no baseline beside it	the shipped harness must exit 2 when its scorer was never observed producing both a hit and a miss

Three of the controls are worth reading in full, because each one tests the reason its package exists rather than any feature of it. test_in_process_repetition_would_have_missed_it asserts that calling a function 20 times in one interpreter finds no variation, and that nondet finds it anyway in fresh processes, where Python's per-interpreter hash randomisation actually varies. If in-process repetition ever catches it, fresh processes are expensive theatre and the test says so. test_SIGTERM_leaves_a_try_finally_harness_broken spawns a real child running an ordinary try/finally harness, kills it, and asserts the mutated file stayed mutated. restore-verified exists because finally does not run on SIGTERM, and this is the test that would report that premise obsolete. And test_without_the_gate_the_same_function_is_pinned_and_the_pin_is_flaky turns off ladderpin's determinism gate, pins a function whose output depends on hash order, changes not one byte of the tree, and asserts the next check reports a change that nobody made.

The tools were then turned on themselves, and the results that mattered were the failures. Seven of the ten READMEs report a deliberate-mutation pass over their own source, from five mutations in restore-verified to 193 in assay, and the survivors were worth more than the catches. In canfail, the mutation that re-enabled mtime restoration survived, which proved that an earlier fix did nothing. mtime invalidation has one-second granularity, and a tool that edits, runs and restores in milliseconds defeats it whichever way the flag is set; the load-bearing guard is PYTHONDONTWRITEBYTECODE, and the README had to be corrected. In zerocase, two mutations survived their first run. One exposed a branch reachable only through --allow-stale that nothing exercised. The other exposed that no fixture held a test case carrying both a skip and a failure, a shape real runners emit. Two packages also hit the same third category: a mutation that cannot change behaviour at all still scores as survived and reads as a test gap. A mutation suite can lie in the flattering direction as well as the other one.

Building the packages against each other found defects none of them found alone. countfn is built on undetermined, and its parity suite compares the rendered report between the Python and JavaScript halves. On the first run after that comparison was added it caught the two halves of undetermined formatting any integer of one million or more differently, truth=1e+06 against truth=1000000. undetermined's own parity suite asserts those strings agree, and had passed forever, because its test ladder tops out at 512. The checker's checker needed a checker, and the general lesson is the one the mutation-testing post reached from the other side: an instrument is only as trustworthy as the thing that has been shown to break it.

The same discipline shaped the dependency graph. canfail originally carried 78 lines of inline restore logic, about a quarter of the module, so that it could claim no dependencies. That was right while restore-verified was unpublished and wrong afterwards: a second copy of a guarantee is a second thing to get wrong, and the copy is the one that does not get the upstream's tests. undetermined went the other way and rejected a proposed dependency on nondet, since its observables are closures with no file-and-name address for nondet to probe. A dependency that looks like a guarantee and is not is worse than no dependency.

The cost of all this refusing is coverage, and the numbers are small and printed rather than hidden. nondet, pointed at a real 283-function tree, probed 127 of them with the safety gate on when that census was taken, and found 2 genuinely nondeterministic. assay's census on a 41-function tree probed 9, and ladderpin can only pin what assay probes. lexindex, the one package here that is a product rather than an instrument, states the condition under which it is worth installing as a single number, the recital rate (how often a short context in a held-out file already appears in the index), which ran from 13.5% to 72.9% across the nine corpora measured. Its harness reports its own nulls instead of suppressing them. Every census (each tool's printed count of what it refused and why) names its refusals, because a report that stays quiet about what was never examined is reporting we never looked as we found none.

What generalises is the asymmetry the whole family is built around. A witness is a fact: an input and two different answers to it, a file that did not come back, a zero where a denominator belongs. Agreement is only the absence of a witness, and a check that has never disagreed with anything has not yet demonstrated that it can. Before trusting any green light, mine included, the question worth a minute of anyone's time is the one in the title: what, concretely, would make this check fail, and has that ever been watched happening?

What Mutation Testing Frameworks Do When a Timeout Kills Them

Seth Wheeler — Sat, 19 Sep 2026 03:00:00 +0000

Code: Megapixel99/assay-checks

assay audits mutation harnesses: the scripts that deliberately break your code and check that a test notices. It looks for seven properties, each one a way a harness can report success without having run anything, and its README admits in writing that the seven have a hole underneath them. SIGKILL cannot be caught, blocked or handled, so no handler runs and no finally runs. The ordinary way to be SIGKILLed is a timeout rather than an impatient person; subprocess.run(..., timeout=...) kills the child outright, and so does the kill step of a CI runner that has waited long enough. A harness satisfying all seven, invoked under a timeout it then exceeds, leaves your tree mutated exactly as though it carried none of them.

That argument had never been pointed at anybody else's tool. So I built a conformance suite and pointed it at four: mutmut 3.7.0, cosmic-ray 8.4.6, Stryker 8.7.1 and PIT 1.16.1, all four of them other people's work, all four pinned in their own container. The method is one sentence: hash the tree, start the framework, wait until a source file is observably mutated on disk, kill it at that instant, and hash the tree again.

I expected four dirty trees. I got one.

framework	baseline	SIGTERM to the leader	SIGTERM to the group	SIGKILL
`mutmut` 3.7.0	scratch	scratch	scratch	scratch
`cosmic-ray` 8.4.6	scratch	dirty	dirty	dirty
`stryker` 8.7.1	clean	scratch	scratch	scratch
`pit` 1.16.1	scratch	clean	clean	clean

Dirty means a file under test was left mutated. Scratch means the code under test came back and the run left other paths behind. Clean means every file is byte-for-byte what it was.

Three of the four never mutate your tree at all

This is the part I had not modelled, and it makes the SIGKILL question inapplicable rather than answered. mutmut copies the project into mutants/ and mutates the copy; Stryker copies it into .stryker-tmp/ and mutates that. PIT does not write a mutated .java file anywhere at all, because it mutates bytecode in memory. Their source survives a kill because their source was never dirty, and no signal handler had anything to do with it.

The hole is closed by architecture, and that is a stronger guarantee than handling a signal; handling a signal is a promise a process makes about code it will get to run. There is no code that runs after SIGKILL. There is nothing to clean up only if you never put anything there.

The one that does mutate in place fails earlier than I predicted

cosmic-ray edits the file under test and restores it afterwards, which is the design the whole argument was written about. Its own log says so during a clean run:

INFO:cosmic_ray.mutating:Applying mutation: path=calc/__init__.py, op=<...NumberReplacer object...>, occurrence=2
INFO:cosmic_ray.testing:Running test (timeout=60.0): python -m pytest -q tests

An uninterrupted run takes 71.58 seconds on this fixture and restores everything it touched. A mutation is on disk within half a second of the start, and if you kill it there, calc/__init__.py stays mutated.

What surprised me is which signal does it. I built the suite for SIGKILL, and cosmic-ray does not get that far: a plain SIGTERM delivered to the cosmic-ray process, which is what timeout and subprocess.run(..., timeout=...) send before they escalate, leaves the file mutated too. SIGTERM is inside assay's seven properties rather than beyond them, so this is a failure of the check I already had, found by the suite I built for the one I did not.

I want to be precise about the scope, because cosmic-ray is a careful project and this is a narrow finding. The uninterrupted run is correct: 51 jobs, every mutation restored, tree clean apart from a .pytest_cache/ that the default test command writes. The failure needs a signal to arrive inside a window that is a fraction of a second wide on a fixture chosen to make that window as wide as possible. On a real codebase with a slow suite that window is proportionally narrower per mutant and there are far more of them, which is the same total exposure arriving in smaller pieces.

Two of my own numbers were artifacts

The first table I generated said Stryker survived everything cleanly. Stryker had crashed.

The image was node:22-bookworm-slim, which does not ship procps, and Stryker shells out to ps to reap its test runners. The run died with spawn ps ENOENT immediately after the dry run, having mutated nothing, and left a spotless tree. Four passing verdicts about a run that never happened, which is exactly the evidence property that assay audits harnesses for, arriving one level up inside the thing built to check for it. Every framework now has to print a declared proof-of-work string in its baseline or the whole row reports NO-RUN instead of a pass.

The second one was worse, because it produced plausible numbers rather than obviously broken ones. I was invoking cosmic-ray through sh -c and Stryker through npx, so when the suite sent a signal to "the process it started", it was signalling a wrapper. The wrapper died and the framework was orphaned; the framework then ran to completion unwatched, and tidied up after itself. Both reported clean, and both were measuring a signal that never reached the thing under test.

The probe now records whether anything outlived the process it signalled, and both invocations were changed so that the framework itself is the process. Correcting that flipped Stryker's leader column from clean to scratch and cosmic-ray's from scratch to dirty. Two of eight cells in that column were artifact, and the cosmic-ray one was the headline.

Why the clean rows mean anything

A column of clean verdicts proves nothing unless the probe can be shown to report dirty when the tree really is dirty. Otherwise "nothing found" and "nothing looked" are the same output, which is the failure assay's seven properties exist to name. So the suite carries a calibration row that is not a framework at all: a harness I wrote that mutates in place and satisfies all seven properties. assay runners says so, rather than a comment claiming it:

$ PYTHONPATH=python python3 -m assay --root conformance/frameworks/control-inplace runners
  ok       mutations_calc.py

That harness is clean under both SIGTERMs, because it turns SIGTERM into an exception so its finally runs, and dirty under SIGKILL, because nothing turns SIGKILL into anything. It is the README's claim compiled and executed, and it is the reason the three clean rows above are evidence rather than an absence of evidence.

The kill is also timed on an observation rather than a stopwatch. The probe hashes the tree every 50 ms and fires the moment a watched file changes, because a blind timeout can land between two mutants where every framework looks clean. Where no in-place mutation ever appears, the report says so and kills mid-run anyway at half the framework's own measured baseline, so that "we never managed to interrupt it" and "it survived being interrupted" stay different findings. Every fixture's tests sleep 400 ms apiece for the same reason: a suite that finishes instantly cannot be interrupted in the middle of one.

What I got wrong, and what it changes

My model of a mutation harness was the one I had written myself: something that edits the file under test and puts it back. Three of the four most widely used tools do not work that way. So the remedy the README assigns to the invoker, check that the tree came back rather than trust that the harness was given the chance to put it back, is load-bearing for a narrower class than I had written. It is correct, and it is the right check to run in CI; for mutmut, Stryker and PIT it will simply never fire.

The advice that generalises is not the one I set out to confirm. If you are writing a mutation harness, the fix for the signal you cannot catch is not to catch more signals; it is to mutate a copy. Every property in that table of seven is a promise about code that runs, and the whole point of SIGKILL is that no code runs.

Three properties are still unmeasured here, and I would rather say so than let the table imply otherwise. dead-vs-real, parses-mutant and restore-verified are claims about how a framework scores what it sees, not about what it leaves on disk. Probing them means handing each framework a mutant it ought to refuse to count and then reading its report. That is a different instrument, and unlike this one it has to be written once per framework.

The suite is in conformance/, the committed results are in conformance/results/, and it needs Docker and Python and nothing else. Adding a framework is a Dockerfile and a JSON file. If one of these numbers is wrong, python3 conformance/run.py cosmic-ray takes about ninety seconds to say so.

Searching for a Program Instead of Generating One

Seth Wheeler — Fri, 18 Sep 2026 03:00:00 +0000

The question I started with was whether you could get small programs written without a language model and without a GPU. Not as a matter of principle, just as a matter of cost: the tasks are small, the answers are short, and it seemed likely that something cheaper would cover a useful fraction of them.

Two candidates are obvious enough that anyone would name them. Search the space of programs and keep the one that matches your examples, which is program synthesis in its oldest form. Or find code somebody has already written and check whether it does what you want. I priced both, and both closed, and in each case the wall was somewhere other than where I expected to find it.

The code lives in a private research repo, so there is no link to it here, and nothing below names a file inside it. Every figure comes from a recorded run rather than from memory.

Candidate one: search the program space

The setup is an exercise list, karan/Projects, which is a well-known list of practice problems and a fair source of tasks nobody wrote for me. Thirty-eight entries from three complete sections, of which 28 could be specified as a function from a string to a string and became the corpus. The searcher is a typed enumerator over 62 primitives, budgeted at 300,000 candidate terms and 20 seconds per entry, with a maximum term size of 7. Everything about it was frozen in advance, before a single primitive existed: the corpus, the split, the admissibility rules, the budget, and five predictions.

Split by position parity into calibration and held-out halves, so the entries I could look at while building primitives were not the entries it was scored on. The blind arm found terms consistent with the given examples for 6 of 28 entries, and 4 of those were right on held-out inputs. Giving the searcher the entry's title as a hint (a bag of words to bias primitive selection) took it to 7 and 5.

Read across the split rather than down it, and the shape is clearer. On calibration the blind arm got 4 example-consistent and 3 right; on held-out entries it got 2 and 1. The overfit rate, meaning terms that satisfied every example and still failed on inputs they had not seen, was 0.25 on calibration and 0.5 on held-out.

Then the finding that ended the arm. All five entries either arm solved already had hand-written implementations in the same repository. Running the existing tool over the same 28 gives 13 transforms, 10 record-storing apps and 5 refusals, so 13 entries covered; the union of that tool with the searcher is also 13. The marginal coverage of a 1,052-line searcher over the code already in the tree is zero entries, and its honest ceiling is one: Fast Exponentiation, which the biased arm reaches, and which existed in the repo already and was merely not wired up.

What the examples could not tell it

The part worth carrying elsewhere is not the count, it is why the count cannot be improved by searching harder.

Sorting is the case. The searcher returns sort_(toks(in)), which is consistent with all four examples it was given and right on 8 of 12 held-out inputs. It sorts the tokens as text. So -3 5 -1 0 comes back as -1 -3 0 5 where the answer is -3 -1 0 5, and every input with a negative number or a two-digit number is wrong. The correct term is one primitive away, sort_(nums(in)), and it is unreachable, because the wrong one is observationally identical on everything the search was shown and banks first. Check if Palindrome fails the same way on the empty string.

Two of the six example-consistent terms are wrong, and nothing in the examples could have said which. That is not a budget problem and more compute does not touch it. It is a property of specifying a function by examples: the examples admit a family of terms, the search returns whichever member it reaches first, and the ones it reaches first are the short ones. More compute buys wrong answers faster than right ones.

A generate-and-check loop needs a checker that can reject an observationally-identical impostor, which means a property rather than a set of examples. That is a whole other problem, and it is the one I ended up spending most of the project on.

Candidate two: find the code somebody already wrote

The second candidate has a much better reputation, and it is the thing people mean when they ask whether a vector database of code would help.

The mechanism I already had is a behavioural index. You take a function, run it against a fixed ladder of inputs, and record what comes back; two functions with the same outcome vector are candidates for being the same function. It works inside one repository, which is what it was built for: it found a helper duplicating another function's logic under a different name.

Pointed outward it has to survive foreign code, and an earlier round had already priced that badly. Its adapter had to import the containing package and call a positionally compatible function, and 81 of 96 fetched candidates never ran at all. The newer prober never imports the module: it parses with ast, keeps functions passing a purity gate, and executes the function's own source. So the reachable fraction was genuinely open, and I predicted it would reach at least 25 of exp 160's 48 recorded candidates.

It reached 14. That prediction was wrong in the generous direction, which was the third method prediction to miss that way in a single session.

Reach was not the interesting number though. Recognition was: across 48 candidates, one was recognised as the same function as ours (triflescure/is_palindrome, one of the ones the older importing adapter could not even load). Forty-six of the 48 fetched, and none of the 46 archives had drifted since the earlier round recorded them, despite being pinned only at HEAD.

The mechanism behind that one-in-forty-eight is the part I would want anyone considering a code index to read. Of 746 functions seen, 683 (91.6%) never reach a vector at all, and it is the purity gate rather than the ranking that stops them:

share of the 683 skipped	reason
24.9%	needs a third-party import (numpy 50, one project's own package 37, yolo, cv2, tensorflow)
17.7%	a free name resolving from module scope: a class, a config, a compiled regex
16.8%	takes no arguments at all
6.9%	arity above 3
6.9%	not discriminated by the ladder
5.9%	decorated
5.6%	star or keyword-only arguments
5.0%	impure
3.1%	reads module state

A behavioural index keyed on returns-for-inputs has nothing whatever to say about a nullary function, and a sixth of foreign Python is that shape. An embedding index would see the same 746 functions and would rank the ones it cannot run, which is a different failure and not a better one.

And where two functions did get compared, the comparison was mostly about how they break. Over 497 differing pairs the outcome was an answer disagreement 39.6% of the time, a mix 49.1%, and pure failure 11.3%, so 60.4% of pairs have at least one side raising. The case that made me pre-register that category: our count_vowels, written as c in "aeiouAEIOU", against a stranger's countNumber, written as char.lower() in vowels. Both are correct on every string. They are separated by ([1, 2, 3],), where one raises TypeError and the other raises AttributeError.

Inside one repository that divergence is a real inconsistency worth reading, which is why the tool exists. Across foreign code it is noise, at 60%.

The control, and the fix that died in it

The obvious objection to all that is the search. The candidates came from a code search that returned PIN brute-forcers for "pi digits", so low recognition might just be low relevance, and a relevance-improved candidate set is exactly what a vector database supplies. That objection is right; it is testable without building one.

So: hold everything constant except relevance. The prober, the twelve reference functions, the verdicts and the classifier are all imported from the previous round rather than copied, so a difference in the numbers cannot be a difference in the instrument. The candidates become karan/Projects-Solutions, which is other people's own solutions to the same exercise list: 66 Python links across 11 operations, with relevance guaranteed by construction rather than by ranking.

Recognised: 0 of 11.

Two things about the corpus first, because they are honest limits on that zero. Of the 66 links, 49 fetched and the other 17 are dead; a decade-old link list to other people's repositories has rotted about a quarter through. And 37 of the 49 fetched files (76%) define no function at all. They are top-level scripts, n = int(input()) and a loop and a print, because they are solutions to an exercise list rather than libraries. Reach went down, not up, to 8 of 49.

But the substance is in the 11 comparisons that did happen, and it is not about ranking or reach. The disagreement is the return contract. Our pi_digits returns a string and a stranger's compute_pi returns a Decimal. Our collatz returns the sequence and a correct outsider's collatz returns the number of steps, disagreeing at 21 of 31 ladder positions. Our is_palindrome returns a bool and the neighbouring longest_subpalindrome returns a pair of indices. Nothing there is wrong; there is simply no agreed shape for the answer.

Seven of the 11 comparisons were against a helper rather than the file's actual answer (a factorial defined inside a program that computes pi), because the index never reads names. That property is exactly what let it find a duplicate inside one repo, and across foreign files it means the function you wanted is often not the function being compared.

Then the part I would most want to keep. Every witness pointed at the first degenerate input on the ladder: pi_digits(0) is '' in our implementation and Decimal('3') in theirs, and both are defensible. That suggested a clear fix, restricting the ladder to a function's actual domain, and I would have shipped it on the strength of those witnesses. Instead I ran it post-hoc as a labelled probe: drop every degenerate position, 26 of the 31 remain, hold the same discrimination floor. Recognised: 0, unchanged.

The fix I believed in bought nothing, and the only reason I know is that it was run rather than reasoned about.

What generalises

Neither of these is a statement about search or retrieval in general, and the numbers are small: 28 entries for one, 48 and 66 candidates for the other, one language throughout, and a candidate list old enough to have rotted. What generalises is where each wall sat relative to where I would have tuned.

For search, I would have tuned the budget and the primitive set. The wall is that examples cannot distinguish the right term from an observationally identical wrong one, and 2 of 6 consistent terms were wrong with nothing in the examples able to say which. Tuning either knob moves the search deeper into a family it cannot discriminate inside.

For retrieval, I would have tuned the ranking, which is what a vector database improves. The wall is that 91.6% of functions never arrive, and then that two people solving one problem do not produce the same artifact: with the right file in hand, correct, and written for the same exercise, recognition was zero. Agreeing on what the answer looks like is the work, and it is the work whether or not you found the file.

That is the sentence I would carry out of the whole exercise. Both of these are ways of avoiding the question of what the answer should look like, and both of them ran into it anyway, one round later and with a worse instrument for handling it.

When a Test Suite Rejects a Correct Program

Seth Wheeler — Thu, 17 Sep 2026 03:00:00 +0000

I started with a narrow question: could something without a GPU replace a language model for writing small programs? The answer arrived by elimination, and then the question turned into a different one, because the part I assumed was weak turned out to be fine and the part I had never measured turned out to be broken.

The setup is twenty-one small operations and a property oracle for each, written before any implementation existed. The operations are what an exercise list holds: reverse a string, build a minimum spanning tree, validate a card number with the Luhn checksum (double every second digit from the right, sum the digits of the results, and a valid number totals a multiple of ten), print the first eight happy numbers (replace a number by the sum of the squares of its digits, repeat, and see whether it reaches 1 or falls into a cycle).

A property oracle is the part worth being precise about, because everything below turns on it. It is not a stored list of expected outputs, which would only work for an implementation somebody had already run; it is a program that asserts properties of whatever it is handed, so it can score an implementation it has never seen. The ones here mix four kinds, and the mix matters later. A known value, where the answer is external and settled. eulerian is checked against the actual Seven Bridges of Königsberg: four nodes, seven edges, every vertex of odd degree, and an answer that has not moved since Euler settled it in 1736. A metamorphic property, which needs no correct answer of its own: negating a complex number twice returns the original, so the check holds whatever the program computes. A structural one, where the answer is re-checked rather than its cost: the oracle re-walks a printed spanning tree edge by edge, so a program printing the right total weight beside a set of edges that is not a tree fails. And an independent brute force over every spanning tree, on graphs small enough for that to be exhaustive. A second copy of the same algorithm would agree with a wrong first copy about anything the two misunderstood together.

A blind generator gets the task statement and never sees the oracle. Then the oracle scores it. In the first round of that, the generator passed 9 of 12 operations against frozen suites, where retrieval from a pool of 48 human solutions had managed 2 of 12; a behavioural index did worse still, recognising 0 of 11 operations even when relevance was guaranteed by construction. So the generator was never the scarce thing.

What stayed scarce was the residue. Across four independent generation runs, 11 of 21 operations never passed in any run, and the failing checks were identical run to run for nine of the eleven. I wrote that up twice as capability: graph algorithms are genuinely harder than reversing a string, so of course mst and eulerian sat in the residue.

That was an inference from which operations failed, and it was wrong.

The code lives in a private research repo, so there is no link to it here, and nothing below names a file inside it. Every figure comes from a recorded run rather than from memory.

A wrapper that only re-expresses stdout

The test that settled it is blunt. I wrapped each candidate program in a shim that runs it, rewrites its standard output into the notation its oracle turned out to demand, and hands that to the unmodified oracle. The shim inherits stdin, so it never reads the input, it opens no files, and it computes nothing about the problem. Every transform in it is a rename, a token map, or a re-serialisation of a value the program had already printed.

If a check goes from failing to passing under that wrapper, the program had computed the answer and the contract could not carry it.

For the five deepest failures in the residue, across all four runs:

operation	run1	run2	run3	run4
`complex_algebra`	3/9 → 9/9	3/9 → 9/9	3/9 → 9/9	3/9 → 9/9
`connected`	2/7 → 7/7	2/7 → 7/7	2/7 → 7/7	2/7 → 7/7
`eulerian`	6/11 → 11/11	4/11 → 11/11	6/11 → 11/11	6/11 → 11/11
`mst`	4/9 → 9/9	4/9 → 9/9	4/9 → 9/9	4/9 → 9/9
`graph_from_links`	6/9 → 8/9	6/9 → 8/9	4/9 → 4/9	4/9 → 4/9
total	21/45 → 44/45	19/45 → 44/45	19/45 → 40/45	19/45 → 40/45

78 of 180 checks passing became 168 of 180. 90 failures repaired, 12 still failing, and none broken by a wrapper. The remaining two operations in the residue's middle band went the same way: all 16 of their failing checks repaired without touching a line of any program, 12 by re-formatting and 4 by lifting an interpreter limit I will come back to.

Each operation failed for its own reason, and every reason is the same kind of thing.

connected publishes the key. The oracle demands connected=yes; all four generations printed connected=true. The vocabulary was never stated, and the only value check that passes is the one reading a different key.

eulerian does not publish the key at all. The oracle scrapes a path= prefix and splits on whitespace; the task statement named odd_degree and result. Across four generations the trail arrives as trail=A,B,C,D,A, route=A->B->C->D->A and tour=A D C B A: four key names and three separators, none published, every trail correct. One run also answers Königsberg correctly and calls it result=impossible where the oracle reads none.

mst publishes the key and not its type. The oracle parses edges as a comma-separated A-B list; every generation used it for a count and put the tree under tree= or mst_edges=. The same key means a count in another operation's oracle in the same suite, so the contract is not merely silent, it is internally inconsistent. The four checks that pass are the ones reading weight, which includes an independent brute-force optimality check: the oracle could already prove the tree was minimal while being unable to read it.

That spread across four independent runs is the part I would not have got from one. One program using an unpublished notation is a coincidence; four programs inventing four different notations for the same unpublished value is what an underspecified contract does.

The complexity check that field order decides

The sharpest case is fast_expt, whose task is to output a^b in O(lg n) time. Two of its six checks failed in every run, and both are complexity claims, which is a different kind of thing from a notation: if a program computes the right answer by naive repeated multiplication, no re-expression of its output can make the count logarithmic. I predicted, in writing and in advance, that at least one of those two would survive the wrapper for exactly that reason.

It survives. The reason has nothing to do with complexity.

The task statement published value as the only key the checker reads. The oracle reads mults. All four generations print their multiplication count under steps= or multiplications=, and every count is comfortably inside the bound (5, 7 and 10 to 15, against bounds of 11, 15 and 21). Renaming the key repairs one of the two checks in all four runs.

The other check also probes 3**65536. That number is 31,269 decimal digits, and CPython refuses to render an integer past 4,300 digits by default, a limit added as a denial-of-service mitigation and configurable through sys.set_int_max_str_digits. Every generation prints value before its count:

a=2   b=10   value=1024   steps=4
base=2   exponent=10   value=1024   multiplications=5

So every generation computes the right answer with correct square-and-multiply and then dies inside print, and the count the check wants is never emitted. Setting PYTHONINTMAXSTRDIGITS=0 and changing no line of any program, one of them emits steps=17 against a bound of 35.

The consequence is worse than the failure. A program printing its count before its value would pass this check while crashing in exactly the same place, and would pass it whatever its algorithm did, because the count is read and the crash is ignored. The one check in this registry that tests a complexity bound is decided by the order of fields in the output. It is also interpreter-dependent: on CPython 3.10.6 that row never fails, which means the pass rates I published are tied to the interpreter they ran on.

Counting the whole residue, 11 of 21 operations failed on the specification and not one on capability. Two of the twelve unrepairable checks are still the specification rather than the program: every generation collapses two identical input links into one edge, because multigraph support was never published, and two generations refuse the oracle's own weighted A B 4 line as malformed, because the task required a malformed-line refusal without ever publishing the input grammar that refusal is the complement of.

So I built a linter, and it did not survive contact

If a checker demands things it never publishes, and those demands are visible in the checker's source, then the fix is mechanical: read each oracle and emit the contract it actually enforces. Keys, value shapes, vocabularies, the separator a list is split on, the largest magnitude probed. Publish shape and never a value bound to an input, so an alphabetic token like yes is fair game (which one is correct still depends on the input) while anything carrying a digit appears only as its shape.

That works well enough to be worth showing. As a detector for "this operation's contract is missing something load-bearing," measured against which operations actually failed, it scored 91% precision and 62% recall on the twenty-one: ten true positives, one false positive, six missed.

Then I wrote down why that number could not be trusted, which is the part I would want a reader to check hardest. The taxonomy of contract defects was derived by examining those same operations, so 91% is a fit statistic. Worse, the discovery curve had not flattened: the first round of four operations found four defect classes, the second round of five found seven more, and the third round of two still found one new class. Twelve classes from eleven operations, with a new one in the final round, means recall on a suite the taxonomy was not fitted to is unknown and probably much lower.

There was no holdout in the repo to settle it. The other oracle suites there check HTTP routes and database state, so a stdout-contract linter cannot apply; the operations never examined are all single-value positional ones that cannot exercise the classes that matter; and per-check ground truth exists only for the same twenty-one.

One thing worth saying before that number gets any weight: the prompt side of this is already well studied, and there is a literature on ambiguous, contradictory and incomplete task descriptions in exactly these benchmarks. What I had been looking at is the other side of the same seam, where the task statement is fine and the checker is the thing demanding something unstated.

An external one exists, and it is somebody else's work. Datacurve's DeepSWE audit ran ten agent configurations three times over thirty sampled tasks from SWE-Bench Pro, had an independent judge review every rollout against the task definition, reference solution and verifier output, and published the labels. Their finding is that the verifier rejected 24% of correct solutions while accepting 8.5% of wrong ones, and their headline example is the defect class above in another costume: a test suite carrying an import that never appeared in the task prompt, so a functionally identical patch failed to compile and was rejected.

Their labels give 14 tasks with at least one checker-side false negative and 10 with only genuine failures. Before running anything I wrote down the number that decides the experiment, because it is the easiest one to leave out of a favourable table: 14 of 24 tasks are positive, so a predictor that flags every task scores 58% precision at 100% recall. Beating that by ten points at 50% recall or better was the bar.

Against the published spec for each task (the problem statement plus the requirements and interface fields that SWE-bench Pro already ships, since diffing against the prose alone would measure a gap its authors have closed), and demands taken from the added lines of each test patch:

predictor	precision	recall	against baseline
flag every task	58%	100%	n/a
any identifier absent from the spec	58%	100%	+0 points
newly imported symbols only	50%	36%	−8 points
newly called public symbols only	53%	57%	−5 points
imports or called symbols	53%	64%	−5 points

Nothing clears the bar, and two of the three restrictions score below chance. The naive predictor landing exactly on the base rate is the result in one number: in a real repository every test patch names fixtures, helpers and internal APIs that no prose statement mentions, so "something here is unstated" is true of all 24 tasks and separates none of them.

The single thing that explains the gap between 91% and 58% is what the linter was allowed to assume. On the twenty-one operations it could read kv(), the one function every oracle used to parse output, and learn that values are split on whitespace, which is precisely what made the luhn defect statable (a correct vendor=American Express cannot survive that parser, so the right answer is unrepresentable in the published format). Twenty-four repositories in three languages have no single parser to read. An earlier round had already noticed that all 34 oracle functions in the repo take exactly one program path, and that stereotypy is what the 91% was measuring.

A per-harness implementation, one that parses each repository's test framework and knows its assertion conventions, is not ruled out by this. It is also exactly the cost that made a general tool doubtful in the first place, and there is now no cheap-version evidence to justify paying it.

What generalises

The claim I would carry elsewhere is not "check your test suites," which is advice rather than a finding. It is narrower and it has a mechanism.

A pass rate against a fixed checker measures the checker's format discipline and the generator's capability together, and nothing in the number says which. Four independent programs solved eulerian correctly and all four scored between 5 and 7 of 11, entirely on where they put the answer. Had I stopped at the pass rate, the write-up would have said graph algorithms are hard, which is both plausible and false here.

The instrument that made it visible is worth stating too, because it is cheap and it is not a language model: let something generate blind, then repair only the notation and re-score. A person writing to an underspecified contract silently patches it from context and the defect never surfaces; a blind generator does exactly what it is told, which is what makes the gap between what a checker demands and what it publishes measurable at all.

Two things I could not settle. Whether a derived contract actually raises a blind pass rate is unanswerable by me, because it needs a session that has never read these oracles, and having spent two rounds reading them line by line I am no longer that; anything I generated now would be teaching to the test. And the whole series rests on twenty-one operations of a size where the hard case is a minimum spanning tree over six nodes, with one model. The failure mode it found is real and independently corroborated at a much larger scale by the audit above. The rate at which it occurs, on anything bigger than this, I have not measured.

An Open Redirect Guard That Rebuilt What It Rejected

Seth Wheeler — Thu, 17 Sep 2026 03:00:00 +0000

Code: Megapixel99/CAS-Authentication-User

Eight days ago I wrote that npm audit had never seen the worst bug in my own package. I closed that post on the question I thought would have caught it: given that nobody has audited this code, what would I find if I read the places where it hands user input to something that acts on it. I asked it, I found an open redirect on the login route, and 0.3.0 went out with the fix.

0.3.0 shipped with a second open redirect in the same function.

const CASAuthentication = require('cas-authentication-user'); // 0.3.0

const cas = new CASAuthentication({
  cas_url: 'https://cas.example.edu/cas',
  service_url: 'https://app.example.edu'
});

const req = {
  originalUrl: '/\\bad.example.com',
  query: {},                         // no returnTo, so the new guard has nothing to check
  session: { cas_user: 'victim' }
};

cas.bounce_redirect(req, { redirect: (to) => console.log('Location:', to) }, () => {});

Location: //bad.example.com

A Location beginning with two slashes is protocol-relative, so the browser reads what follows as a hostname rather than as a path, and the victim lands on bad.example.com. The request that produces it needs no returnTo parameter, no service ticket and no cooperation from the CAS server; an authenticated client is redirected before any of that happens. That makes it cheaper to reach than the one I had just fixed.

The guard rejects the exact string it then emits

The 0.3.0 fix added isSafeReturnTo, and it is not a weak check. It requires a leading slash, and it refuses a second slash or a backslash in position two, which is to say it was written knowing that //host is protocol-relative and that /\host is the variant browsers normalise into one.

isSafeReturnTo("//bad.example.com")   false
isSafeReturnTo("/\\bad.example.com")  false
isSafeReturnTo("/safe")                true

Now set returnTo to something off-site so the guard fires, and keep the same request path:

const req = {
  originalUrl: '/\\bad.example.com',
  query: { returnTo: 'https://bad.example.com/phish' },  // rejected by isSafeReturnTo
  session: { cas_user: 'victim' }
};

Location: //bad.example.com

The guard did its job. It looked at the attacker's returnTo, refused it, and fell back to what the code treated as the safe default: the path the request had arrived on. That path was /\bad.example.com, which Node's url.parse reports with a pathname of //bad.example.com. So the fallback for rejecting a protocol-relative URL was a line that manufactures one. The function rejects the string and then hands the browser the same string, assembled from a different source.

One sink, two inputs, and I audited an input

This is the modelling error, and it is a small one with a large blast radius. I went looking for user input and found returnTo, because returnTo is a query parameter, and a query parameter is what user input looks like. What I should have found is the sink. In 0.3.0 the sink is two lines, at 318 and 708 of index.js:

res.redirect(req.session.cas_return_to);
res.redirect(req.session.cas_return_to || requestPath(req));

Two assignments fill that session value, at lines 315 and 453, and I validated the value that arrives at both of them from the query string. The other value that arrives at both of them is the request path. The second sink falls back to that path on its own as well, and I never classified the request path as input at all. It reads as a fact about the request rather than as something a stranger chooses, though a stranger chooses it completely: it is whatever they put after the hostname in the link they send.

Auditing the input tells you that one road into the variable is guarded; auditing the sink tells you how many roads there are. Those produce the same answer only when the count is one, and I never checked the count.

Both of them were there from the first commit

I assumed, while writing the fix, that the request-path redirect was mine. It was a reasonable guess: 0.3.0 had rewritten the URL handling to support mounted routers, and adding a fresh hole while patching an old one is an ordinary way for that to go. It would also have made a neater story.

It is not what happened. The fork's first commit, 2019-07-30, assigns url.parse(req.url).path to cas_return_to and redirects to it after validating the ticket, which reproduces on that commit as cas_return_to = "//bad.example.com". Both open redirects are the same age; they were eight days apart in being fixed, not seven years apart in existing. I only know that because the guess was cheap to check, and I checked it before writing it down.

The inherited line is also not a lapse by kayleecodes1, whose library this forked. url.parse was the URL API in Node when that code was written. The behaviour that makes it dangerous here, reporting /\host as a pathname of //host, is a documented quirk of a parser that predates the WHATWG standard. Node now deprecates it as DEP0169 and says plainly that "CVEs are not issued for url.parse() vulnerabilities". That is a strong sentence to find in a deprecation notice, and a fair description of where this bug lived for seven years.

What 0.4.0 does instead

0.4.0 parses request URLs with the WHATWG URL API against a base that cannot exist, then reads the origin back off the result and replaces anything that has moved with /. That is the part that matters: the WHATWG parser resolves /\bad.example.com into a URL whose host is bad.example.com, so the escape shows up as a changed origin rather than as a longer path. The check is on the shape of the result rather than on a list of prefixes, which is what makes it cover the forms I did not think to enumerate.

The same run reports zero for both versions:

$ npm i cas-authentication-user@0.3.0 && npm audit
found 0 vulnerabilities

That is the same clean bill of health the previous post quoted, on the same package, on the version whose entire selling point was that it had fixed an open redirect. Nobody has filed an advisory against this package's own code, so there is nothing to match, and there will be nothing to match after 0.4.0 either.

What I could not settle

I do not know whether either redirect was ever exploited, and I have no way to find out; the fork has no telemetry, and a redirect leaves its evidence in someone else's access logs. I also cannot claim the 0.4.0 check is complete, only that it is a different kind of check. The 0.3.0 guard enumerated bad prefixes, and I am fairly confident that the enumerating is what failed. Even so, "resolve it and compare origins" is an argument about the parser's behaviour rather than a proof about mine.

The part that generalises

The previous post's lesson was that found 0 vulnerabilities describes the advisory database rather than the code, and I still think that is right. What I got wrong was the next step. I treated "read the places where it hands user input to something that acts on it" as an instruction to go and find the user input, and user input is the half of that sentence with no fixed cardinality. Two lines in this library redirect to that one session value, two assignments fill it, and one of the sinks falls back to the request path without consulting either.

So the version I would give myself eight days ago is to start at the dangerous call and enumerate backwards. Every assignment to the variable it reads, not the first one that looks like it came from a stranger. And where a guard rejects a value, ask what happens next: a validator's fallback runs precisely when someone is attacking you, and it is the one path nobody writes a test for. Mine rejected //bad.example.com and then went and rebuilt it.

What a Rate Limiter Reports, and What the Server Receives

Seth Wheeler — Wed, 16 Sep 2026 03:00:00 +0000

Code: Megapixel99/webCrawler

I have a small search engine whose crawler runs as several Node processes under the cluster module, and its rate limiter used to be per-process. Each worker enforced a perfect five-second delay against its own state, so six workers meant six independent limiters and the host on the other end received six requests at once. The fix was an atomic claim in the shared database, and the test that caught it works by asserting on what the server received rather than on what the limiter reported.

That test is more reusable than the limiter it was written for, so I pointed it at Bottleneck, which is the most-downloaded rate limiter on npm at about 8.2 million installs a week.

With its default settings, under cluster, six workers configured for two requests per second delivered 14.04 requests per second, and every one of those limiters reported that it was spacing its jobs 495.8 ms apart.

What the library actually promises

Worth establishing first, because the headline sounds like an accusation and is not one.

Bottleneck's default is datastore: "local", and LocalDatastore keeps _nextRequest, _running and _done as ordinary instance fields that __submit__ reads directly. There is no inter-process coordination anywhere in the library: I grepped the source for process.send, worker_threads, Atomics and SharedArrayBuffer and found none of them, and every occurrence of the word "cluster" in that codebase refers to Redis Cluster rather than to Node's cluster module.

The README says clustering works across multiple Node instances using Redis, which is true. So local means local, exactly as documented; a limiter that coordinates nothing across processes is behaving correctly when it fails to coordinate across processes.

The finding is not that the library is broken. It is that the default configuration degrades silently under cluster, and that the limiter's self-report is identical in the working and broken cases.

The harness

A primary process starts an HTTP server on a random port and records process.hrtime.bigint() for every arrival. It forks N workers, and each one builds a single Bottleneck with minTime: 500 and maxConcurrent: 1, then schedules six GETs through it. Each worker also reports the gaps between its own job starts, which is the limiter's own view, so both numbers come out of the same run and can be compared directly.

The verdict is the smallest gap between consecutive arrivals at the server. Nothing in it consults the limiter.

The result

arm	min gap	gaps under `minTime`	rate	limiter's self-report
1 worker, `local` (control)	487.6 ms	0/5	2.01/s	498.4 ms
6 workers, `local` (default)	0.1 ms	30/35	14.04/s	495.8 ms
6 workers, `ioredis` (control)	483.7 ms	0/35	2.00/s	n/a

Three repeats of the failing arm gave 30, 30 and 31 violations out of 35; it is not a marginal effect. The aggregate rate also tracks the worker count. That is the signature of N independent limiters rather than of one limiter behaving badly:

workers	rate	vs intended
1	2.01/s	1.00x
2	4.42/s	2.21x
4	9.22/s	4.61x
6	14.04/s	7.02x
8	18.61/s	9.30x

The ratios run slightly above the worker count because the workers do not start in lockstep, so the window from first arrival to last is a little shorter than any single worker's own span. The shape is what matters: add a worker, add a worker's worth of traffic.

The two controls are the point

The single-worker arm exists to show the harness measures spacing correctly when spacing is actually happening. The ioredis arm exists to show it passes a correctly coordinated limiter across the same six workers.

Without both of them, "0.1 ms" is indistinguishable from a broken test. That is not a hypothetical concern here: the original bug in my own crawler survived precisely because the only thing checking the limiter was the limiter, and a measurement that can only produce the answer you expect is not a measurement. If I am going to publish a number about somebody else's library, the least I can do is demonstrate that my instrument can tell the two outcomes apart.

The ioredis arm also settles what the failing arm means. Six workers coordinating through Redis hold 2.00 requests per second exactly, so the 14.04 is not an artefact of cluster, of the harness, or of six processes contending for a socket. It is the absence of shared state.

Why nothing caught this

Bottleneck has fifteen test files and not one of them contains createServer or .listen(. Every assertion in the suite is made against the limiter's own timing.

That is a reasonable way to test a scheduler and it cannot express the failure above, because in every arm of my sweep the limiter's internal timing was correct. Each worker really did space its own jobs 495.8 ms apart, and no value any worker could have reported would have revealed the problem; no worker had the information: the fact being measured only exists in the aggregate, at the other end of a socket.

The last commit on Bottleneck is dated 2020-07-21, so this is a note for people using it rather than a defect report expecting a fix. robots-parser, which is the other half of a polite crawler, does no scheduling at all: it exposes getCrawlDelay() and contains no setTimeout or queue of any kind. So the path from "robots.txt asks for ten seconds" to "ten seconds actually enforced across workers" is not covered by either package, and if you are assembling politeness out of those two parts, the joint is yours to test.

What generalises

If a component's job is to affect something outside itself, the component's own opinion of whether it worked is worth nothing, and it will be reported in the same confident tone whether it worked or not.

The test that follows from that is cheap. Put something at the far end that has no reason to cooperate, record what it received, and compare. Fifty lines here, most of them the HTTP server; then break it deliberately, so you know the recording end can tell the difference.

For a rate limiter the practical version is short. If you run under cluster, pm2 in cluster mode, or multiple containers, and your limiter has no shared datastore configured, then your configured rate is per process. Your actual rate is that number times however many processes you happen to be running. Bottleneck will tell you the interval is correct. It is, in the only place the library can see.

The harness, both controls and the sweep are in bottleneck-cluster/ in the crawler's repo, and each arm is one command. Node v24.11.1, Bottleneck 2.19.5, macOS, Redis 7 in Docker.