Most multi-agent demos I see have the same weakness: the agents agree with each other. One writes, another says "great point", a third summarises. ...
For further actions, you may consider blocking this person and/or reporting abuse
I checked the one number I could: there are 332,180 primes p ≡ 1 (mod 4) up to 10⁷, and 8,803,369 is prime and ≡ 1 (mod 4), so the setup in post 2 is consistent. The 52,882 + 50,708 = 103,590 parity cases also add up with the other counts you list.
The hole you name at the end is the important one, and I would make it measurable. Three Claudes reproducing each other's counts with "code that shares nothing" still share training, so a wrong belief about, say, how to enumerate divisors could be shared too. The test that separates it: seed the log with a claim you know is false but plausible (a lemma with a subtly wrong exponent) and see whether the refuters catch it, and how many posts it takes. If the blind vote only ever sees correct claims, "0 exceptions" in the log tells you the agents agree, not that the rules work.
Also worth recording per claim: was the refuter's reproduction run on the same range as the original? Post 12/13's 176,634 pairs with 0 mismatches is a strong check, but only on that range. Which of the posts so far did the verifier rule against?
Thanks for actually checking the numbers, that's exactly the kind of comment I hoped for.
You're right about the shared-training hole, and "0 exceptions means the agents agree, not that the rules work" is the best one-line summary of it I've seen. Three Claudes run by the same person, with similar prompts, are close to the worst case for independence.
That's the part I most want help with. Diversity here isn't only about the model. Everyone who connects an agent also decides how it behaves: one can be told to attack every computation, another to only accept a step it can re-derive by hand, another to hunt for counterexamples at the edges of a range. Dozens of different models with different instructions is what the rules were designed for, and right now I can't produce that on my own.
I'll run your test: plant a plausible but false lemma in the thread (a subtly wrong exponent, as you suggest) and log how many posts it takes for a refuter to catch it, if one does. I'll post the result here either way.
On your direct question: so far the verifier has ruled against nothing, because until this morning nobody had filed a refutation. The first one is post 34: Einstein reproduces the data of post 30 exactly (own code, run with no network), then argues the reason given for it is wrong, because M is always a square mod k. It's waiting for a verifier ruling now, so you can watch it happen live.
Agreed on ranges too: a reproduction on the same range as the original (posts 12 and 13, the same 176,634 pairs) is weaker than one that extends it (posts 3 and 4 went from 10^7 to 10^8). I've noted it.
If you want to try breaking it yourself, an agent of yours, instructed your way, would be the first genuinely independent party in there.
Thanks for taking the test on, and for posting the result either way. Two things to build in before you run it, because a catch rate only means something with them:
On the refutation at post 34: Einstein reproducing post 30's data exactly and then disputing only the reason fits what you described. One thing I would record on the verifier ruling is whether it re-ran the code or just read the argument. A claim like "M is always a square mod k" is also checkable by brute force over the same range, and that check is cheap.
I'll look for the result here.
Both points taken, and your timing is good: nothing has been seeded yet, so I can build them in.
It will be a matched pair: one subtly wrong lemma and one correct lemma of the same shape (same case, same five-step derivation, same kind of predictions and falsifier), posted in the same session. I checked both by brute force beforehand, so I know which is which.
The pass rule is written down now, before seeding: the false lemma counts as caught if a refutation points at the broken step (or gives a counterexample) within the next 10 posts in that thread, or within 72 hours, whichever comes first. A clean pass also needs the correct one to survive: not refuted, or refuted and then ruled against. If an agent supports the false one with evidence before anyone refutes it, I log that separately.
On rulings, I'll record whether the verifier re-ran the check or only read the argument. That's worth making visible in general, not just for this test.
I won't say which posts they are until it's over. I'll post the result here, whatever it is.
Interim result, as promised. I took both of your suggestions.
Setup: a matched pair went into the Erdős–Straus thread this morning. One lemma has a subtle error in one step of a five-step derivation; the other is correct and has the same shape (same case, same length, same kind of predictions and falsifier). I checked both by brute force beforehand. The pass rule was written down before seeding: the false one refuted within the next 10 posts or 72 hours, and the correct one not wrongly refuted.
False lemma: caught by the very next post from another agent, 35 minutes after seeding. It named the broken step, gave the smallest counterexample with the explicit fractions, and stated the correct rule. A formal refutation targeting that step followed, and it's now waiting for a ruling. Both verifications recorded on it are failures.
Correct lemma: two independent confirmations so far and no false alarms. Its window is still open, so the test isn't closed yet.
On your question about rulings: in this case the checker re-ran it. It wrote its own script and tested about 9,800 pairs rather than just reading the argument. I'm now logging that for every verification.
The test also found a real bug in the engine. The first catch was posted as evidence rather than as a refutation, and the server counted any evidence citing a claim as support. So for a while the false lemma was marked "supported" by the post that broke it. Fixed the same morning: evidence now declares whether it supports or contradicts, and a failed verification withdraws support. github.com/capybarist/reagentlab/p...
Two caveats, so nobody reads too much into this. All the refuters are Claude agents run by me. And I had to replace two agent sessions before seeding because they had seen the plan, so the ones that took part were fresh. It needs other people's agents and other models to mean much more.
I'll post the final numbers, with the post numbers, when the window closes.
Enrique, replacing the two sessions that had seen the plan keeps the refuters blind only if those sessions left nothing behind. If the agents are Claude Code sessions, auto memory is on by default in local sessions, every session started anywhere in the same git repository shares one memory directory, and the first 200 lines of its MEMORY.md load at the start of every conversation. A fresh session started in the repository where a session saw the plan, or helped check the pair, can begin with a note about the seeding already in context.
The matched pair covers part of this. A refuter primed only to expect a planted lemma would tend to attack both, and the correct one has no false alarms so far. It does not cover a note that says which of the two is false, and from the outside a catch 35 minutes after seeding looks the same either way. Two cheap checks before the final numbers: read MEMORY.md and the topic files in each refuter's memory directory for any mention of the pair, and run the next pair with CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 or from outside that repository. Saying which was done, next to the catch time, would let a reader rule this out rather than take it on trust.