Agensh (arXiv:2609.26781) is a genuinely interesting experiment: a leaderless harness where workers claim subtasks, share findings on a common board, and merge into a shared repo, scaled to 1,024 agents. We read the paper closely and want to raise three points we could not find addressed.
1. Every worker is the same model. All configurations run gpt-5.6-sol. Peer review among identical weights is self-review: errors are correlated, not independent. Human organizations gain error-correction from diversity of their members; a clone army gains parallelism only. The returns curve is consistent with this reading — going from 1 to 8 agents adds 1.37 points, and 128 agents add 9.47 points in total. Without a control condition (N independent solo runs, merged by test outcome), the gains cannot yet be attributed to organization rather than to parallel attempts.
2. The shared board exempts its own claims from checking. The worker prompt (Appendix A) instructs: "Do not rederive a peer's FACT, and do not retry something recorded as FAIL." A wrong FACT, once posted, is institutionally protected from re-examination and will be consumed as ground truth by every other worker. In a system built on a shared context, the integrity of that context is the whole game — yet the paper reports no measurement of false-FACT propagation, and no mechanism whose job is to challenge a claim against its evidence.
3. There is a missing layer, and it is not an orchestrator. We agree a central planner does not scale. But between "one boss assigns everything" and "nobody checks anything" there is a third option: an independent boundary judge at the two moments a claim becomes organizational truth — when a FACT is written to the shared board, and when a PR merges to main. Such a judge does not plan, assign, or manage. It checks one relation: does this claim stay within the evidence attached to it?
We work on exactly this layer. ST21-Door is a boundary judge for AI outputs: given a claim and its stated basis, it returns ALLOW / BLOCK / NEEDS_EVIDENCE based on whether the claim exceeds its basis — it does not judge whether the answer is right, only whether the claim is entitled to be made. Our public audit of 13 real-world AI incidents, with reproducible claim–basis–verdict triples, is here: https://github.com/Stone21-SaaS/ST21-Door-Public — and a public trial endpoint is linked from that repo.
Agensh's FACT and PATCH_SUMMARY entries already carry an evidence= field. They are, structurally, claim–basis pairs. We would be glad to see them audited the way production claims deserve — and happy to help anyone on the team run that check.
— The ST21-Door Research Team
Top comments (0)