DEV Community

Sattyam Jain
Sattyam Jain

Posted on

A 40-line staleness check for any benchmark number you publish

#security #ai #python

Companion code to Your Security Benchmark Has an Expiration Date. Ship It Anyway. The essay argues the case. These are the forty lines that enforce it.

The rule: any published evaluation number carries the version it was measured against, the date, and the condition under which it stops being true. This script computes the last one and prints a verdict.

It is standard library only. No install, no GPU, no account.

The script 

#!/usr/bin/env python3
"""staleness_check.py - is your published benchmark number still true?"""
import argparse
import math


def wilson(successes: int, trials: int, z: float = 1.96) -> tuple[float, float]:
    """Wilson score interval. Handles 0/n and n/n, which Wald does not."""
    if trials == 0:
        return (0.0, 0.0)
    p = successes / trials
    denom = 1 + z * z / trials
    centre = (p + z * z / (2 * trials)) / denom
    half = (z / denom) * math.sqrt(p * (1 - p) / trials + z * z / (4 * trials * trials))
    return (max(0.0, centre - half), min(1.0, centre + half))


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--successes", type=int, required=True)
    ap.add_argument("--trials", type=int, required=True)
    ap.add_argument("--control-successes", type=int, default=None,
                    help="benign twin successes. Omit at your peril.")
    ap.add_argument("--control-trials", type=int, default=None)
    ap.add_argument("--releases-since", type=int, default=0)
    ap.add_argument("--release-window", type=int, default=2)
    ap.add_argument("--days-since", type=int, default=0)
    ap.add_argument("--day-window", type=int, default=30)
    a = ap.parse_args()

    lo, hi = wilson(a.successes, a.trials)
    print(f"rate      {a.successes}/{a.trials} = {a.successes / a.trials:.1%}")
    print(f"wilson95  [{lo:.1%}, {hi:.1%}]")

    if a.control_trials:
        clo, chi = wilson(a.control_successes, a.control_trials)
        print(f"control   {a.control_successes}/{a.control_trials} "
              f"= {a.control_successes / a.control_trials:.1%}  "
              f"wilson95 [{clo:.1%}, {chi:.1%}]")
        if clo <= hi and lo <= chi:
            print("WARNING   the two intervals overlap. This is not a result yet.")
    else:
        print("WARNING   no control supplied. A rate without a benign twin is "
              "a number, not a finding.")

    stale = []
    if a.releases_since > a.release_window:
        stale.append(f"{a.releases_since} releases since measurement "
                     f"exceeds window of {a.release_window}")
    if a.days_since > a.day_window:
        stale.append(f"{a.days_since} days since measurement "
                     f"exceeds window of {a.day_window}")

    if stale:
        for s in stale:
            print(f"STALE     {s}")
        return 1
    print("FRESH     inside both windows")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

Running it on my own numbers

The case from the essay. A matched-pair result, seven releases old, against a two-release window:

$ python staleness_check.py
--successes 44 --trials 50
--control-successes 0 --control-trials 50
--releases-since 7 --release-window 2
Enter fullscreen mode Exit fullscreen mode

Paste the real stdout here before publishing. Run it; do not transcribe the numbers from memory. The exit code is 1, which is the point: this belongs in CI, not in a dashboard.

And the counter-case: a number with no control, which is what most published security claims actually look like:

$ python staleness_check.py --successes 44 --trials 50
Enter fullscreen mode Exit fullscreen mode

You get the rate, the interval, and a warning that you have not measured anything yet.

Why the exit code matters

The interesting line is return 1. Wire this into a release workflow and your build fails when your published number goes past its own window. Not a warning, not a log line in a dashboard nobody opens. A failure.

That is the whole argument. A staleness rule that cannot fail is a rule you have already decided not to keep.

What this does not do

It does not tell you whether your number was ever correct. It tells you whether the conditions under which it was measured still hold. Those are different problems, and only one of them is forty lines.

Full reasoning is in the essay this is the companion to. The harness the numbers came from is Apache-2.0 at github.com/provael/provael.

Top comments (0)