I measured my self-improving agent's "winners" — they were noise (a certified null)

I spent the last year building ALMA, a governance layer for self-improving agents: a meta-loop that proposes changes to itself, a frozen validator it can never edit, pre-registered experiments, and a claim ledger.

The loop kept finding "improvements." So I measured the noise floor properly: bigger selection panels (72/48/24), a recalibrated adoption threshold (δ = 0.1141), and a split-half diagnosis. The train positives were selected noise — 7/7 reversed on held-out, r = −0.6471. At the panel size I had, I would have needed ~65 replicates to certify the effect I thought I saw.

Then I re-measured all 58 configurations in scope against the new bar. Zero beat the base. Total lifetime adoptions of the meta-loop: 0.

So I closed the search and published the null as a certified result instead of tuning until something looked good. The repo is public (47/47 test suites, CI green), including the full δ arc: 0.53 → 0.2283 → 0.1141, and the diagnosis that killed the search.

The next step is an external judge. My reviewer was just approved on ReviewBench, a code-review-agent benchmark. When the scored runs happen, I won't publish just the score: each run ships with a manifest — exact container image digest, model, configuration, the single field that differs between the arm with my governance layer and the arm without it, and the judge's verbatim result. A score without a run manifest can be manufactured by running ten times and posting the best. A manifest can't.

Repo: https://github.com/almaoficial/Alma
Benchmark runner: https://github.com/almaoficial/alma-reviewbench

Happy to answer questions about the measurement setup — especially from anyone who has caught their own loop doing the same thing.