Code: Megapixel99/undetermined
Plenty of libraries fit a curve to measurements and hand you back a number. undetermined hands back a number with the error bar it was decided by, plus an explicit undetermined list carrying a reason on each entry. It will put an observable on that list rather than fit a plateau to a drift. The README's demo adapter flips seeded coins at a ladder of input sizes (8, 32, 128, 512; a rung is one size). heads counts heads among truth flips, so the fair coin's factor of 2 is what settles: it comes back as 1.9978 +/- 0.0032, with the rung its plateau started from. flat ignores truth entirely, and comes back UNDETERMINED, with the reason that no run of 3 rungs agrees and the constant is still moving at the top of the ladder. The second line is the point, and it is also the falsifiable part. A committed pair of tests asserts that the drifting observable lands on the list while the settling one in the same run does not. A tool that always produced the first line would be useless and would still pass every test that checks it produces one.
The scenario is any constant you believe a program has (bytes per record, operations per element, a coin's factor of 2) that has to be measured rather than read off. You supply an adapter that can run the thing at a controllable input size; the library never knows what program it is looking at:
from undetermined import characterize
class Coin:
def truths(self):
return [8, 32, 128, 512] # the input sizes: a ladder
@property
def observables(self):
return {"heads": heads, "flat": flat} # each is (truth, seed) -> number
report = characterize(Coin(), trials=2500)
report["undetermined"] # ['flat'], with the reason in notes
pip install undetermined and npm install undetermined are the same tree at the same version. It refuses three ways of being confidently wrong. An observable that ignores its seed raises immediately, because the fitter averages over different seeds, and a mean over noise still has a standard error, still forms a ladder, and can still plateau. Every guard downstream compares against that error, so a broken adapter does not produce a wrong-looking answer. It produces a confident one. A constant that never settles is reported with its reason, since a plateau requires three consecutive rungs agreeing within two combined standard errors. And a choice between observables that is not earned is refused too. An observable must vary at least three times its own measurement error across instances to be called informative, and must beat the runner-up by the same factor to be chosen. Even the degenerate case is a raise rather than a shrug. An adapter with one observable leaves the library no choice to be shown making.
The rule underneath all of it is to compare against the noise, never against the size. A spread only means something in units of the error on the thing that spread, and dividing by the magnitude instead is how a large number gets mistaken for a real one.
That rule is also where the library was wrong, and the 0.2.0 release notes the failure in a way I want to quote at length because of one clause. Every error bar used to be Type A, the scatter of repeated draws, sd/sqrt(N). An observable that answers the same number every time has no scatter, so its standard error was zero, the ladder-builder dropped the rung, and the report said the constant could not be determined. It said that about a quantity it had measured exactly, in the same words it uses for a quantity with no constant at all. Everything the library had been measured on was a stochastic simulation, so the case never came up. Pointed at real systems it is the common case. And the clause that stings: a test in the repository asserted the old behaviour. The defect was not merely unnoticed, it was pinned. It surfaced because countfn, a sibling package built on this one, hit it for every deterministic operation count. That package wrote the finding into its README rather than working around it quietly.
The fix is the metrologist's combined standard uncertainty, u = sqrt(u_A² + u_B²), where u_B is granule/sqrt(12) and the granule is the resolution the observations are reported at. u_B is deliberately not divided by sqrt(N). Repeating a deterministic measurement does not buy resolution, and an error bar that shrank when you looped would let any constant be made significant by asking twice. The granule derivation errs fine on purpose. A coarse granule widens every error bar, and wide error bars are how a search flattens a drift that was never constant; the alternatives (a GCD of the values, the spacing between them) can only err coarse, so neither is used. The fixtures ship the control for the fix itself. exact and drifting are both deterministic, exact is now recovered, and drifting is still UNDETERMINED in both halves. A fix that widened error bars until the first one worked would have flattened the second.
Two halves ship from one tree, at one version, on PyPI and npm, and the thresholds are asserted identical down to the explanatory strings. That forced a shared number formatter. %g and toPrecision(6) agree only on integers below 10⁶ and non-integers in [1e-4, 1e6), and %.1f and toFixed(1) round halves in opposite directions. The shared rule renders an exact integer in full (a rung at 16777216 is not clarified by calling it 1.67772e+07) and rounds half away from zero. Both halves are pinned to expected strings independently, since two halves that had drifted together would still agree with each other.
One dependency decision is worth recording because it went the unfashionable way. A nondet edge was considered for the reproducibility precondition and rejected. nondet addresses functions as FILE::NAME, this library's observables are closures with no such address, and wiring it in would have meant a fake path or a check that never ran. A dependency that looks like a guarantee and is not is worse than no dependency, so the precondition is implemented natively in both halves, in the same call that would have needed the guarantee.
Top comments (0)