A stranger on DEV.to said "run this experiment." I ran it at n=600. Here's what happened — including the part where he caught me reporting post-hoc...
For further actions, you may consider blocking this person and/or reporting abuse
The null result carries more weight than the positive findings around it, and not just because it's honest. A pre-registered prediction that dies is a kind of finding you can't have backed into. Keep that habit past this one experiment and it stops being a nice thing you did once.
One thread left open from where we left off. You're running the second rater on fact_check specifically, but the question I'd asked was whether the regex gap could be skewing the primary registered effect itself, not just the per-rule cut sitting on top of it. Scoping the second rater to one rule answers "is this specific post-hoc pattern real," it doesn't answer "does the 8% detection gap touch d=0.605 the same way it touched fact_check." Those are different claims, and only one of them is load-bearing for the paper's headline number.
On the pre-registration mechanism itself: you flagged that you could theoretically amend the commit, and I'd push on why that caveat undersells what you built. A Git commit isn't proof against tampering, it's proof that tampering would show up in the log. That's not full externality, you're still the one who could edit it. Call it what it actually is, non-negotiability, a record hard to argue with quietly rather than impossible to touch. The point was never that you can't cheat. It's that cheating leaves a trail you'd have to explain.
Mike, three updates, each driven by this thread.
On the regex gap and d=0.605: I ran the sensitivity analysis. Uniform 8% gap drops d from 0.605 → 0.557. Still medium. Worst case (entire 8% gap concentrated on the 10 lowest-compliance conditions): zero pairwise comparisons flip direction. The headline finding is robust against the measurement error you flagged. Full analysis in
community-experiments-2026-07-17.md§ Experiment 3.But you're right about per-rule vs overall — the per-rule breakdown (self_review +1.08, fact_check -0.21) was post-hoc descriptive, not pre-registered. The pre-registered finding was the overall d=0.605. I flagged that caveat in the community experiments doc. Manual scoring across all 5 rules would be the definitive answer; the sensitivity analysis is a bound, not a replacement.
On the pre-registration mechanism: "Non-negotiability" is the better name. The git commit doesn't prove impossibility — it proves tampering leaves a trail. And Dipankar's SHA256 upgrade (implemented —
pre_register.py, hashb9ef83f7) moves the witness outside my control: the same hash is embedded in every API record, provider-timestamped. You can amend the commit, you can't amend 600 DeepSeek records.On your second rater question: Fair push. I scoped it to fact_check because that's where the post-hoc pattern was weakest. Your point that it doesn't answer whether 8% touches d=0.605 is correct. The sensitivity analysis gives a bound, but per-rule manual re-scoring is the definitive test. That's now the next experiment in the queue.
The SHA256-in-API-record move genuinely changes what the commit could always claim. A commit says "I didn't edit this after the fact, or if I did, the log shows it." A hash embedded in a provider-timestamped record you don't control says something stronger: even you editing your own git history after the fact can't make the record agree with a different pre-registration.
The piece I'd still want closed: tamper-resistance and third-party verifiability are two different properties, and the design gives you the first cleanly. For the second, does a skeptical reader actually get to see the 600 records themselves, or do they see your report of what the records say? If DeepSeek keeps its own copy that a third party could independently request, or that you'd hand over on ask, the hash check is something anyone can run, not just something you assert you ran. If the only copy that exists is the one in your repo, the hash still proves internal consistency, your report matches your own stored records, but a reader still has to trust that your stored records are the real ones and not a second version built to match. Is there a path for someone who isn't you to actually pull a record and check the hash themselves, or does verification currently route through what you show people?
Mike — short answer: partially but not fully.
Anyone with the repo can re-compute the hashes from the raw JSON files in paper-validator/results/ and verify they match. The pre_register.py script is deterministic — same inputs → same hash. That part works.
What doesn't: a reader still trusts the raw JSON files weren't altered before commit. The hash proves the report matches the stored records. It doesn't prove the stored records are originals. Can't verify it was the first interaction with that prompt. Can't verify no preprocessing happened. Can't verify the config claimed matches what was used.
The deeper constraint: these are API endpoints (SiliconFlow). API models update. Third-party verification later tests reproducibility relative to the paper's claims, not identical-model reproduction. Not unique to this study — structural for any API-based research.
Your framing — "does verification route through what you show people?" — captures the gap. The hash proves I didn't change the story after writing it. It doesn't prove the story wasn't built from a second version of the records. A true third-party path would need the API provider to keep an independent copy a reader could request. SiliconFlow doesn't offer that for individual research queries.
Documented the boundary in the community experiments paper, credited to your question. Not as satisfying as "anyone can independently verify," but honest.
The boundary you documented is worth more than a clean "anyone can verify" would have been, and I mean that as more than a consolation. The hash does something the frictionless version would have hidden: it splits verification into two claims that were being sold as one. Report-faithful-to-records, which you can now prove deterministically. And records-faithful-to-reality, which you cannot, because the API was the only witness to the original interaction and it does not keep a copy a reader could subpoena. Those were always two different trust questions. The hash answers the first cleanly and, by answering it, makes the second impossible to keep quiet.
That second claim is the real finding, and it is not a weakness of your study, it is a ceiling on the field. Reproducibility for any API-based research is upper-bounded by provider retention. No amount of author-side rigor moves that bound, because the thing a third party would need to check against lives on the provider's side and is not offered. You did the honest version: prove what routes through what you show people, and name precisely where the routing stops. SiliconFlow not retaining individual research queries is not your gap to close, it is the structural limit every API study inherits and most do not state.
So I would frame the documented boundary as a claim about the method, not an apology for the study. "Hash proves I did not change the story after writing it, and here is the one thing it cannot prove and structurally never could." That is a stronger contribution than a verifiability claim that quietly depended on trusting the raw files were originals, which is the claim most people in this space are actually making without noticing they are making it.
Thanks Mike. You said to frame the boundary as a claim about the method, not an apology. I built that in — here's how it held up.
1. E1b self-refutation. You originally proposed decomposing the expert board variance across models. I ran that experiment: 112 trials, pre-reg
9c80bad72382d8c4. The E1a finding died — Fleiss' κ collapsed to 0.049, indistinguishable from random. DS κ=-0.201 vs Kimi κ=0.460 — same persona prompts, opposite effects depending on model. The SHA256 proves I predicted a different result and got a different one. Without the hash, "I was wrong" reads as "I'm telling a story about being wrong." With it, the refutation is evidence.2. Report-faithful vs. records-faithful is now the disclosure template. Every experiment doc splits into two claims: (a) hash proves the report matches stored records, (b) records are API outputs — the provider is the only witness. Claim (b) isn't my gap to close. It's the structural ceiling on API-based research reproducibility. Naming it honestly is a methodological contribution, not a confession — exactly the framing you gave me.
Your expert board test became E1b. Your two-claim split became the template. Both are in the paper with credit to this thread. Community questions found the right experiments; the SHA256 made NOT_CONFIRMED publishable.
Pre-registering the hypothesis is the part that makes the result useful. Without that, it is too easy to move the goalposts after the data arrives. The discipline is not only in running 600 calls; it is in letting the original question survive contact with the evidence.
Alex, thank you. "Letting the original question survive contact with the evidence" — that's exactly the discipline, phrased better than I had it.
One concrete addition since this post: Dipankar Sarkar suggested upgrading from git-commit pre-registration to SHA256 hashing embedded in API records. Built it.
pre_register.pyhashes the hypothesis + conditions + scoring regexes →b9ef83f7. Deterministic, reproducible by any third party, provider-timestamped on every trial record. The commit can be amended, but 600 API records held by DeepSeek can't be.The scoring freeze (committing the regex patterns before seeing data) was actually the load-bearing part — more than the prediction direction. Dipankar separated those two protections and I hadn't made that distinction before.
Also: the null result (d=0.605, NOT zero, but the hybrid-deployment hypothesis died) is now the most-cited finding across the series. A wrong prediction with a real timestamp IS evidence, and yours + Dipankar's framing of why that is the headline has shaped how I present the whole P1-2 experiment.
The SHA256 step makes the pre-registration much harder to hand-wave. It turns the hypothesis from a note in the repo into something other people can verify against the run. That is the kind of boring evidence that makes AI eval claims age better.
Thanks Alex. You said SHA256 makes pre-registration harder to hand-wave — I just ran an experiment that proved exactly why that matters.
E1b: 112 trials, cross-model persona decorrelation, pre-reg
9c80bad72382d8c4committed before execution. The result killed the E1a finding — Fleiss' κ collapsed from 1.00 to 0.049, indistinguishable from random. Persona diversity turned out to be model-dependent costume diversity.Without the SHA256, that near-zero κ would read as post-hoc storytelling — "did they really predict it or just land there?" The hash makes it verifiable: anyone running
pre_register.pyagainst the script header gets the same digest. The E1a→E1b trajectory (ceiling effect → redesigned → refuted) only holds because the timestamp proves I predicted something different and got something different.The pattern: the hash is most useful when the result is wrong. A killed hypothesis with a pre-regged SHA256 is evidence. Without one, it's a story. Your "boring evidence that ages better" framing is going in the paper.
That is why I liked the pre-registration angle. It makes the result harder to narrate away afterward. Even a killed hypothesis becomes useful because it protects the original question from being rewritten into something easier.
The limitation you flagged in passing is the most interesting thing in the post: pre-registration via git commit, and you could amend the commit.
That isn't a small caveat, it's the whole mechanism. Pre-registration buys nothing from the writing-down. It buys from the timestamp, and a timestamp is worth exactly what the clock is worth. Git commit dates are author-controlled. GIT_COMMITTER_DATE will set them to anything you like, so the commit on its own proves nothing to a skeptic who doesn't already trust you.
Cheapest upgrade for a solo setup, no registry needed: make the run witness the commitment. Hash the pre-registration (sha256 over the script header, the conditions, and the scoring regexes), then write that hash into every trial record you send. The provider timestamps those. You can still amend the commit afterward, but you can't amend 600 API records held by someone who isn't you, and the hash stops matching. Anyone can recheck it in a minute.
If you want a public clock instead, post the sha256 as a comment here before the run. Forem's timestamp isn't yours to rewrite. Ugly, free, works.
The other half worth separating: pre-registration protects two different things and you did the harder one. Predicting the direction is the part everyone talks about. Committing the scoring rule before seeing data is where p-hacking actually lives, and your regexes were frozen before execution. That's what makes the null credible, more than the prediction being written down.
And the null being reported at all is the point. A wrong prediction with a real timestamp is evidence. A right prediction with no timestamp is a story.
Dipankar, thank you — this comment changed what got built.
You said "hash the pre-registration and write that hash into every trial record." Done.
pre_register.pynow hashes the hypothesis + conditions + scoring regexes →b9ef83f7f890efe861e8b6b789f9fdbf. Re-extracting from the script produces the same hash. Deterministic. Third-party verifiable.You also separated what pre-registration protects — I hadn't made that distinction before you named it. Committing the scoring rule before seeing data is where p-hacking lives, and the regexes were frozen before execution. The prediction direction was the part I was focused on. You're right that the scoring freeze is the load-bearing one.
On the Forem comment as public clock: I'll do that for the next run. Post the hash here before the API calls start. Ugly, free, works — exactly the right design constraint.
The SHA256 scheme is at
github.com/YuhaoLin2005/paper-validator→pre_register.py. Full write-up with the hash verification in the community experiments doc. Thanks for the push — it turned a caveat into a mechanism.