A clean sample can sound stronger than it is. Someone checks a handful of items, finds no exceptions, and the sentence wants to become broader than the evidence: the process worked, the control operated, the population was clean.
That sentence is where I stopped trusting the product unless the math had the same status as the API. If a compliance audit sampling tool emits a confidence claim, the confidence procedure is production code. It needs adversarial tests. A formula in a notebook is too far away from the thing users read.
This post is about the bound in core/bounds.py, the inverse sample-size question in the same module, and the evaluation gate in eval/coverage.py and eval/run.py. The tool reports a one-sided upper confidence bound on the number of failures in a finite population. Then it tries to prove that bound covers at the confidence it claims.
1. The number is a frontier, not a fact
The bound answers a narrow question: after drawing n items from a population of N and observing k exceptions, what is the largest true failure count D still compatible with the sample at the requested confidence?
That wording matters. The tool is not estimating the true number of failures. It is stating what the sample rules out. A clean sample does not mean the population is clean. It means failure counts above the frontier are no longer supported by the observed draw at the selected confidence.
The implementation uses the hypergeometric distribution because audit sampling draws without replacement. The same item cannot be selected twice. That is a small detail in code and a large detail in interpretation when the sample is a meaningful fraction of the population, which is exactly the situation where finite-population correction starts to matter (MeasuringU).
DECISIONS.md has the blunt version of the tradeoff. At N=1,000 with n=25, the exact finite-population bound is 11.1 percent; the binomial comparison is 11.3 percent. Close enough that the distinction feels academic. At N=64, the gap opens as the sampling fraction rises: at n=5 the binomial is 7 percent high, at n=25 it is 45 percent high, at n=48 it is 94 percent high, and at a full census of 64 it reports 4.6 percent where the true answer is zero.
That last case decided the design. If I examine every item in a finite population, the upper bound on unobserved failures cannot stay above zero after a clean census. The binomial approximation has forgotten that the population can be exhausted.
def exact_upper_bound(N: int, n: int, k: int, confidence: float = 0.95) -> Bound:
"""What a sample of n from N, showing k exceptions, actually supports."""
_validate(N, n, k, confidence)
D = _max_failures(N, n, k, confidence)
return Bound(
N=N,
n=n,
k=k,
confidence=confidence,
max_failures=D,
max_failure_rate=D / N,
rule_of_three=(3.0 / n) if (k == 0 and n > 0) else None,
binomial_max_failure_rate=_clopper_pearson_upper(n, k, confidence),
)
I kept the binomial value in the returned model as a comparison, never as the answer. That made the approximation visible without letting it drive the verdict.
The other choice in this function is less obvious: the result carries N, n, k, and confidence alongside the bound. core/models.py says the rule directly: no bare float crosses a signature. A number without its inputs is exactly the failure mode this tool is trying to avoid.
2. The hypergeometric inversion is the product behavior
The core search lives in _max_failures. It inverts the probability statement instead of sampling possible worlds. For each possible true failure count D, the observed exception count K has a hypergeometric distribution. The reported bound is the largest D for which observing at most k exceptions is still probable enough.
The implementation depends on monotonicity: adding more failures to the population can only make small observed counts less likely. That turns the search into an exact binary search over integer failure counts, rather than a heuristic over percentages.
if n == 0:
# Zero draws exclude nothing. Every item could be a failure.
return N
if k == n:
# Every draw was an exception. P(X <= n) == 1 for every D.
return N
alpha = 1.0 - confidence
lo, hi = k, N # D = k always satisfies: P(X <= k | D = k) == 1
while lo < hi:
mid = (lo + hi + 1) // 2
if _cdf_at_most_k(k, N, mid, n) >= alpha - _ALPHA_TOL:
lo = mid
else:
hi = mid - 1
return lo
The edge cases are part of the contract. With zero draws, the sample excludes nothing. If every draw is an exception, every possible failure count remains compatible with observing at most n exceptions, so the upper bound is the whole population.
The middle is the frontier. For fixed N and n, every observed k cuts the lattice of possible true failure counts into allowed and ruled-out values. The product prints the frontier, not the hidden truth.
flowchart TD
population[Finite Population N] --> draw[Draw n Without Replacement]
draw[Draw n Without Replacement] --> observed[Observed Exceptions k]
observed[Observed Exceptions k] --> inversion[Hypergeometric Inversion]
inversion[Hypergeometric Inversion] --> frontier[Reported Upper Bound D]
frontier[Reported Upper Bound D] --> claim[Allowed Claim]
population[Finite Population N] --> lattice[All True Failure Counts D]
lattice[All True Failure Counts D] -.-> inversion[Hypergeometric Inversion]
binomial[Binomial Approximation] -.-> wrongFrontier[Forgets Finite Population]
alwaysN[Always Return N] -.-> vacuous[Always Covers But Says Nothing]
inversion[Hypergeometric Inversion] ==> exactPath[Exact Finite Population Bound]I folded the bad alternatives into the same diagram because they fail for different reasons. The binomial approximation can be too loose when the sampling fraction rises. Always returning N would pass a naive coverage check, because it covers everything. The exact finite-population inversion has to satisfy both constraints: cover at the stated confidence and still say something.
3. The inverse question is where product pressure shows up
Once the tool can say what a sample supports, the next question is predictable: what sample size would support a stronger sentence?
That inverse question also belongs in core/bounds.py. It uses the same finite-population bound, only flipped around. Instead of asking, “given N, n, and k, what is the maximum supported failure rate,” it asks what draw size would be needed for a target cap under an assumed exception rate.
I am careful with that framing because this is where tools start to lie politely. A target cap is not a measurement. An assumed exception rate is an input. If those are treated as facts, the sample-size panel becomes a confidence costume for planning assumptions.
The model boundary helps here. In core/models.py, Bound validates internal consistency: k cannot exceed n, n cannot exceed N, the bound cannot fall below the observed exceptions, and the maximum failures cannot exceed the population. That is plain defensive programming, but in this domain those checks are also argument hygiene.
A product that emits statistical language has two outputs. There is the value on the screen, and there is the sentence a human will say after reading it. The inverse sample-size path exists because the second output matters. If the current sample cannot support the stronger sentence, the tool should state the cost of that sentence rather than rounding the current evidence upward.
4. Coverage is a gate, not a chart
The evaluation runner treats coverage as the first measurement. eval/run.py describes four measurements, but coverage runs first and stops the run if it fails. The target is never adjusted to match the result.
That is the right shape for a statistical product. Detection rates, draws to first exception, and prior sensitivity are useful after the confidence procedure is valid. Before that, they are decoration.
eval/coverage.py checks the confidence claim two ways. The exact path enumerates outcomes analytically. The Monte Carlo path replays the same quantity through the random draw mechanism. They are not redundant. Exact enumeration tests the math. Simulation tests that the implementation path agrees with the math.
def exact_coverage(N: int, n: int, D: int, confidence: float = NOMINAL) -> float:
"""P(true D falls inside the reported bound), summed over all outcomes."""
ks = np.arange(0, n + 1)
pmf = hypergeom.pmf(ks, N, D, n)
covered = np.array(
[exact_upper_bound(N, n, int(k), confidence).max_failures >= D for k in ks]
)
return float(pmf[covered].sum())
This is the cleanest test in the project. For a fixed N, n, and true failure count D, every possible observed k is finite. So the test sums the probability mass for the outcomes where the reported bound contains the true D.
There is no sampling error in that number. If it falls below the nominal confidence, the procedure does not cover. The product is wrong.
The Monte Carlo version uses NumPy’s hypergeometric draw and then runs the same bound calculation on the observed exception counts. The default is 10,000 runs with seed 0.
def mc_coverage(
N: int,
n: int,
D: int,
confidence: float = NOMINAL,
runs: int = MC_RUNS,
seed: int = 0,
) -> float:
"""The same quantity by simulation, through the real draw path."""
rng = np.random.default_rng(seed)
ks = rng.hypergeometric(ngood=D, nbad=N - D, nsample=n, size=runs)
bounds = {
int(k): exact_upper_bound(N, n, int(k), confidence).max_failures
for k in np.unique(ks)
}
The dictionary cache is a small detail, but I like it. A simulation can draw the same k many times, and the bound for that k does not change. Cache the unique observed counts, then compare repeated outcomes to the same computed frontier.
The readme states the result before anything else: verdict passed. Minimum coverage observed was 0.9500 against nominal 0.9500, across 5,943 exhaustively enumerated configurations. The worst configuration was N=64, n=25, D=16, at 0.9505. The largest disagreement between exact and simulated coverage was 0.0039.
Those numbers are not marketing. They are the release condition for every sentence that depends on the bound.
5. A passing coverage test can still be useless
A malicious implementation can pass a lower-bound coverage check by returning N for every input. It will always contain the true failure count, because the true count cannot exceed the population.
That is the trap in testing confidence procedures. “Covers at least 95 percent” is necessary. Alone, it rewards vacuity.
The suite adds an anti-vacuity ceiling. The readme states it as a test: minimum coverage must not rise above 0.96. A bound that always returned N would cover at 100 percent and say nothing, so that implementation should fail.
This is one of those tests that looks strange until you need it. Most test suites only reject undercoverage. Here, overcoverage can also be a bug because the product is supposed to say the strongest true sentence supported by the evidence. Too safe becomes uninformative.
There is a second guard: shrink the bound by one and confirm coverage breaks. That catches the opposite fake. Without it, the coverage test could be passing while failing to detect that the frontier is load-bearing. If reducing every reported upper bound by one still passed, the original test would not be sharp enough to prove the implemented boundary.
The two guards create pressure from both sides. The bound cannot be loose enough to say nothing, and it cannot be tightened without losing the claimed confidence.
6. Ground truth stays outside the estimator
The simulator needs hidden failure rates. The estimator must never see them.
That boundary is explicit. simulation/draw.py is the only module permitted to read Stratum.true_failure_rate. Its docstring says why: everything under core/ estimates, while the simulation knows the answer. If an estimator could read ground truth, the evaluation would be theatre.
The same file says the boundary is enforced three ways: parse every file under core/ and api/ for the identifier, assert that no estimator module imports the simulation package, and perturb every true rate in the fixtures while checking that estimator output does not move by a single digit.
I care about this boundary more than I expected to when I started. Statistical code has an especially ugly failure mode: it can look more correct after it cheats. If ground truth leaks into the estimator, the bounds become beautifully calibrated to information the auditor never had.
The HTTP surface follows the same rule. server/main.py says every response is a public projection. Stratum carries simulator ground truth, so nothing of that type is serialized. The browser receives projections, never the domain object that knows the answer.
That separation costs some convenience. You write more models. You pass more values around. You make tests about identifiers and imports, which feels crude until the first time it prevents a silent category error.
7. The formula was the easy part
The finite-population bound is compact. The production problem is larger: keep the approximation visible but out of the verdict, carry the inputs with every output, invert the distribution over integer counts, prove coverage by enumeration, replay it through simulation, reject vacuity, and make ground truth unreachable from estimators.
I do not think of that as “extra validation” around the math. It is the math becoming a product surface.
A confidence number is a promise about repeated behavior. Once that promise appears in software, the test suite has to attack the promise itself, because the user will build a sentence from the number whether the code earned it or not.
🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series
Top comments (0)