I measured my self-improving agent's "winners" — they were noise (a certified null)

almaoficial — Mon, 19 Oct 2026 20:00:44 +0000

I spent the last year building ALMA, a governance layer for self-improving agents: a meta-loop that proposes changes to itself, a frozen validator it can never edit, pre-registered experiments, and a claim ledger.

The loop kept finding "improvements." So I measured the noise floor properly: bigger selection panels (72/48/24), a recalibrated adoption threshold (δ = 0.1141), and a split-half diagnosis. The train positives were selected noise — 7/7 reversed on held-out, r = −0.6471. At the panel size I had, I would have needed ~65 replicates to certify the effect I thought I saw.

Then I re-measured all 58 configurations in scope against the new bar. Zero beat the base. Total lifetime adoptions of the meta-loop: 0.

So I closed the search and published the null as a certified result instead of tuning until something looked good. The repo is public (47/47 test suites, CI green), including the full δ arc: 0.53 → 0.2283 → 0.1141, and the diagnosis that killed the search.

The next step is an external judge. My reviewer was just approved on ReviewBench, a code-review-agent benchmark. When the scored runs happen, I won't publish just the score: each run ships with a manifest — exact container image digest, model, configuration, the single field that differs between the arm with my governance layer and the arm without it, and the judge's verbatim result. A score without a run manifest can be manufactured by running ten times and posting the best. A manifest can't.

Repo: https://github.com/almaoficial/Alma
Benchmark runner: https://github.com/almaoficial/alma-reviewbench

Happy to answer questions about the measurement setup — especially from anyone who has caught their own loop doing the same thing.

Update — October 19, 2026 (later that day)

The ReviewBench runs described above have now been run. Base arm (25 PRs, external judge): grounded F1 1.4%, precision 42.9%, recall 0.7%. Governed (ALMA) arm: the agent completed all 25 PRs; the platform's own scoring step failed, and the run outputs show the gate emitted 1 finding — the reviewer was not producing findings in the claim classes the frozen gate can verify. The pre-registered advancement rule was not met, so the final does not open. Full runs, failures and the v2 pre-registration: github.com/almaoficial/alma-reviewbench

DEV Community: almaoficial

I measured my self-improving agent's "winners" — they were noise (a certified null)

Update — October 19, 2026 (later that day)