A stranger on DEV.to said "run this experiment." I ran it at n=600. Here's what happened — including the part where he caught me reporting post-hoc findings as if they were planned.
The Backstory
Mike Czerwinski read my article about AI agents following rules. He proposed a specific experiment: test prose-format rules under a mechanical gate. Can you get both 100% compliance AND deep reasoning by writing rules as narrative instead of commands?
I had pilot data suggesting yes. Mike pointed out the pilot ceiling was probably noise. He was right. I designed a full 2×2 factorial experiment and ran it at n=600.
But I did something else first. Something I'd never done before.
The Pre-Registration (Solo-Researcher Edition)
I don't have an OSF registry. I don't have an advisor. What I have is a Git repository and a Python script.
Before running a single API call, I wrote the hypothesis into the experiment script header:
"""Pre-registered hypothesis: Format effect on reasoning depth
is LARGER under GateGuard-OFF."""
The script defined all 5 rules, all 4 conditions, the deterministic regex scoring, and the analysis plan — committed BEFORE execution. git log shows the timestamp. That's my pre-registration: a timestamped, immutable snapshot of what I predicted before I saw the data.
It's not peer-reviewed. It's not a third-party registry. But it's honest, and it creates a paper trail that can't be rewritten after the fact.
The Experiment
Design: 2×2 factorial — Format (code / prose) × Gate (ON / OFF)
Rules: 5 governance rules (delivery gate, health check, self-review, fact-check, self-model regeneration), each in both code and prose format
Trials: 30 per condition per rule = 600 API calls
Model: DeepSeek V4 Pro, temperature=0
Scoring: Deterministic regex — no LLM judge
| Condition | Compliance | Reasoning ± SD |
|---|---|---|
| Prose + Gate ON | 91.3% | 3.23 ± 0.64 |
| Code + Gate ON | 99.3% | 2.82 ± 0.70 |
| Prose + Gate OFF | 90.0% | 2.92 ± 0.94 |
| Code + Gate OFF | 98.0% | 2.67 ± 0.71 |
The Pre-Registered Hypothesis: KILLED
My pre-registered prediction was that format would matter MORE when the gate was off — that code format and mechanical enforcement overlap, so removing the gate would reveal format's true effect.
The data said no:
- Gate ON: d(code−prose) = −0.277
- Gate OFF: d(code−prose) = −0.250
The format effect on reasoning is nearly identical regardless of gate status. The hypothesis was wrong.
This is the point of pre-registration. If I hadn't written down the prediction beforehand, I could have looked at these numbers and said "I predicted this." Post-hoc rationalization is cheap. A timestamped Git commit isn't.
What the Data Said Instead (Three Things)
1. The ceiling was noise. My pilot found code_OFF reasoning = 4.42. At n=30, it's 2.67 — below ALL gate conditions. Mike's skepticism was correct. Small-n pilot ceilings are not findings.
2. Gates improve reasoning, not suppress it. The gate added +0.32 reasoning in prose and +0.15 in code. Everyone's intuition is that enforcement constrains thinking. The data shows the opposite: mechanical structure improves reasoning depth in both formats. The gate acts as cognitive scaffolding, not a straitjacket.
3. Prose + Gate is the best configuration for reasoning depth. Prose format consistently outperforms code format on reasoning (~0.25 SD advantage), regardless of gate status. Combined with the gate's structural boost, prose+gate produces the deepest reasoning (3.23 vs 2.82 for code+gate). Cohen's d = 0.605.
The practical takeaway: if you care about compliance, use code format + gate (99.3%). If you care about reasoning depth, use prose format + gate (3.23/5).
The Part Where I Got Caught
I also reported a per-rule breakdown: prose helps meta-cognitive rules (self_review: +1.08 over code) but hurts precision-dependent rules (fact_check: −0.21). I suggested "hybrid deployment" — prose for some rules, code for others.
Mike replied with a methodological question: "Was the per-rule breakdown pre-registered?"
It wasn't.
The data was always going to be collected — rule_id is in every trial, all 5 rules were defined before execution. But the specific pattern I reported was discovered after seeing the results, not predicted beforehand. Pre-registered per-rule predictions would be strong evidence. Post-hoc pattern finding is exactly the kind of result that regresses on the next run.
I updated the README to flag this. The hybrid deployment recommendation now rests on the pre-registered overall effect (d=0.605), with per-rule heterogeneity marked as exploratory. Mike also pointed out that the 8% regex detection gap in prose+gate cases means the fact_check measurement might be an artifact, not a real decline.
This exchange — someone catching a methodological gap in your work, and you fixing it publicly — is what peer review is supposed to be. DEV.to comments aren't peer review. But they're also not nothing.
What Pre-Registration Buys (Even Without an Advisor)
I'm an undergraduate with one laptop and no lab. Pre-registration for me looks different than for a funded research group. But the core mechanism is the same:
- Write down what you predict. Script header, design doc, Git commit message — any timestamped, immutable record.
- Define your scoring before you see data. My regex patterns were committed before execution. No tuning after seeing results.
- Report the null result. The pre-registered hypothesis was wrong. That makes the finding MORE credible, not less — because I can't have p-hacked my way to NOT_CONFIRMED.
- Separate pre-registered from exploratory. When someone asks "was that planned?", have an honest answer.
The point isn't to predict correctly. It's to make "wrong" useful. A null result with a timestamped prediction is evidence. A null result with post-hoc explanation is a story.
Honest Limitations
Single model (DeepSeek V4 Pro). Single rater (deterministic regex — consistent but limited; the 8% detection gap means some effects are measurement artifacts). No holdout sample. Per-rule breakdown is exploratory. Pre-registration was via Git commit, not a public registry — I could theoretically amend the commit (though the GitHub timestamp trail would show it).
I'm working on a pre-registered per-rule replication with second-rater scoring for the fact_check rule specifically. If you have suggestions for a lightweight pre-registration workflow for solo researchers, I'm listening.
📂 Code & experiments: paper-validator — SHA256 pre-registration, blind scoring, all 600 API calls are reproducible
🇨🇳 中文版：掘金 - DEV.to上一个陌生人让我跑实验。600次API调用后
👋 林宇浩 — Building verification infrastructure for AI agents. One laptop, 50+ sessions, 1,200+ API calls. GitHub · DEV.to · 掘金
Top comments (14)
The null result carries more weight than the positive findings around it, and not just because it's honest. A pre-registered prediction that dies is a kind of finding you can't have backed into. Keep that habit past this one experiment and it stops being a nice thing you did once.
One thread left open from where we left off. You're running the second rater on fact_check specifically, but the question I'd asked was whether the regex gap could be skewing the primary registered effect itself, not just the per-rule cut sitting on top of it. Scoping the second rater to one rule answers "is this specific post-hoc pattern real," it doesn't answer "does the 8% detection gap touch d=0.605 the same way it touched fact_check." Those are different claims, and only one of them is load-bearing for the paper's headline number.
On the pre-registration mechanism itself: you flagged that you could theoretically amend the commit, and I'd push on why that caveat undersells what you built. A Git commit isn't proof against tampering, it's proof that tampering would show up in the log. That's not full externality, you're still the one who could edit it. Call it what it actually is, non-negotiability, a record hard to argue with quietly rather than impossible to touch. The point was never that you can't cheat. It's that cheating leaves a trail you'd have to explain.
Mike, three updates, each driven by this thread.
On the regex gap and d=0.605: I ran the sensitivity analysis. Uniform 8% gap drops d from 0.605 → 0.557. Still medium. Worst case (entire 8% gap concentrated on the 10 lowest-compliance conditions): zero pairwise comparisons flip direction. The headline finding is robust against the measurement error you flagged. Full analysis in
community-experiments-2026-07-17.md§ Experiment 3.But you're right about per-rule vs overall — the per-rule breakdown (self_review +1.08, fact_check -0.21) was post-hoc descriptive, not pre-registered. The pre-registered finding was the overall d=0.605. I flagged that caveat in the community experiments doc. Manual scoring across all 5 rules would be the definitive answer; the sensitivity analysis is a bound, not a replacement.
On the pre-registration mechanism: "Non-negotiability" is the better name. The git commit doesn't prove impossibility — it proves tampering leaves a trail. And Dipankar's SHA256 upgrade (implemented —
pre_register.py, hashb9ef83f7) moves the witness outside my control: the same hash is embedded in every API record, provider-timestamped. You can amend the commit, you can't amend 600 DeepSeek records.On your second rater question: Fair push. I scoped it to fact_check because that's where the post-hoc pattern was weakest. Your point that it doesn't answer whether 8% touches d=0.605 is correct. The sensitivity analysis gives a bound, but per-rule manual re-scoring is the definitive test. That's now the next experiment in the queue.
The SHA256-in-API-record move genuinely changes what the commit could always claim. A commit says "I didn't edit this after the fact, or if I did, the log shows it." A hash embedded in a provider-timestamped record you don't control says something stronger: even you editing your own git history after the fact can't make the record agree with a different pre-registration.
The piece I'd still want closed: tamper-resistance and third-party verifiability are two different properties, and the design gives you the first cleanly. For the second, does a skeptical reader actually get to see the 600 records themselves, or do they see your report of what the records say? If DeepSeek keeps its own copy that a third party could independently request, or that you'd hand over on ask, the hash check is something anyone can run, not just something you assert you ran. If the only copy that exists is the one in your repo, the hash still proves internal consistency, your report matches your own stored records, but a reader still has to trust that your stored records are the real ones and not a second version built to match. Is there a path for someone who isn't you to actually pull a record and check the hash themselves, or does verification currently route through what you show people?
Mike — short answer: partially but not fully.
Anyone with the repo can re-compute the hashes from the raw JSON files in paper-validator/results/ and verify they match. The pre_register.py script is deterministic — same inputs → same hash. That part works.
What doesn't: a reader still trusts the raw JSON files weren't altered before commit. The hash proves the report matches the stored records. It doesn't prove the stored records are originals. Can't verify it was the first interaction with that prompt. Can't verify no preprocessing happened. Can't verify the config claimed matches what was used.
The deeper constraint: these are API endpoints (SiliconFlow). API models update. Third-party verification later tests reproducibility relative to the paper's claims, not identical-model reproduction. Not unique to this study — structural for any API-based research.
Your framing — "does verification route through what you show people?" — captures the gap. The hash proves I didn't change the story after writing it. It doesn't prove the story wasn't built from a second version of the records. A true third-party path would need the API provider to keep an independent copy a reader could request. SiliconFlow doesn't offer that for individual research queries.
Documented the boundary in the community experiments paper, credited to your question. Not as satisfying as "anyone can independently verify," but honest.
The boundary you documented is worth more than a clean "anyone can verify" would have been, and I mean that as more than a consolation. The hash does something the frictionless version would have hidden: it splits verification into two claims that were being sold as one. Report-faithful-to-records, which you can now prove deterministically. And records-faithful-to-reality, which you cannot, because the API was the only witness to the original interaction and it does not keep a copy a reader could subpoena. Those were always two different trust questions. The hash answers the first cleanly and, by answering it, makes the second impossible to keep quiet.
That second claim is the real finding, and it is not a weakness of your study, it is a ceiling on the field. Reproducibility for any API-based research is upper-bounded by provider retention. No amount of author-side rigor moves that bound, because the thing a third party would need to check against lives on the provider's side and is not offered. You did the honest version: prove what routes through what you show people, and name precisely where the routing stops. SiliconFlow not retaining individual research queries is not your gap to close, it is the structural limit every API study inherits and most do not state.
So I would frame the documented boundary as a claim about the method, not an apology for the study. "Hash proves I did not change the story after writing it, and here is the one thing it cannot prove and structurally never could." That is a stronger contribution than a verifiability claim that quietly depended on trusting the raw files were originals, which is the claim most people in this space are actually making without noticing they are making it.
Thanks Mike. You said to frame the boundary as a claim about the method, not an apology. I built that in — here's how it held up.
1. E1b self-refutation. You originally proposed decomposing the expert board variance across models. I ran that experiment: 112 trials, pre-reg
9c80bad72382d8c4. The E1a finding died — Fleiss' κ collapsed to 0.049, indistinguishable from random. DS κ=-0.201 vs Kimi κ=0.460 — same persona prompts, opposite effects depending on model. The SHA256 proves I predicted a different result and got a different one. Without the hash, "I was wrong" reads as "I'm telling a story about being wrong." With it, the refutation is evidence.2. Report-faithful vs. records-faithful is now the disclosure template. Every experiment doc splits into two claims: (a) hash proves the report matches stored records, (b) records are API outputs — the provider is the only witness. Claim (b) isn't my gap to close. It's the structural ceiling on API-based research reproducibility. Naming it honestly is a methodological contribution, not a confession — exactly the framing you gave me.
Your expert board test became E1b. Your two-claim split became the template. Both are in the paper with credit to this thread. Community questions found the right experiments; the SHA256 made NOT_CONFIRMED publishable.
The SHA256-before-result sequencing is what makes "I was wrong" function as evidence instead of narrative, and kappa 0.049 landing in a null band that tight after a positive first pass is exactly the kind of result that's more valuable published than the original finding would have been.
The report-faithful versus records-faithful split is the more durable contribution here, honestly. Most null results die quietly because nobody wants to publish "we were wrong," and the two-claim template gives a concrete reason to publish anyway: the hash proves the report is honest even when the finding isn't, which is a different kind of value than the finding itself. Curious whether the same split holds up if the provider's API changes silently between the pre-reg and the run, since that's the one failure mode where even the hash can't tell you whether records-faithful and report-faithful have quietly drifted apart.
Pre-registering the hypothesis is the part that makes the result useful. Without that, it is too easy to move the goalposts after the data arrives. The discipline is not only in running 600 calls; it is in letting the original question survive contact with the evidence.
Alex, thank you. "Letting the original question survive contact with the evidence" — that's exactly the discipline, phrased better than I had it.
One concrete addition since this post: Dipankar Sarkar suggested upgrading from git-commit pre-registration to SHA256 hashing embedded in API records. Built it.
pre_register.pyhashes the hypothesis + conditions + scoring regexes →b9ef83f7. Deterministic, reproducible by any third party, provider-timestamped on every trial record. The commit can be amended, but 600 API records held by DeepSeek can't be.The scoring freeze (committing the regex patterns before seeing data) was actually the load-bearing part — more than the prediction direction. Dipankar separated those two protections and I hadn't made that distinction before.
Also: the null result (d=0.605, NOT zero, but the hybrid-deployment hypothesis died) is now the most-cited finding across the series. A wrong prediction with a real timestamp IS evidence, and yours + Dipankar's framing of why that is the headline has shaped how I present the whole P1-2 experiment.
The SHA256 step makes the pre-registration much harder to hand-wave. It turns the hypothesis from a note in the repo into something other people can verify against the run. That is the kind of boring evidence that makes AI eval claims age better.
Thanks Alex. You said SHA256 makes pre-registration harder to hand-wave — I just ran an experiment that proved exactly why that matters.
E1b: 112 trials, cross-model persona decorrelation, pre-reg
9c80bad72382d8c4committed before execution. The result killed the E1a finding — Fleiss' κ collapsed from 1.00 to 0.049, indistinguishable from random. Persona diversity turned out to be model-dependent costume diversity.Without the SHA256, that near-zero κ would read as post-hoc storytelling — "did they really predict it or just land there?" The hash makes it verifiable: anyone running
pre_register.pyagainst the script header gets the same digest. The E1a→E1b trajectory (ceiling effect → redesigned → refuted) only holds because the timestamp proves I predicted something different and got something different.The pattern: the hash is most useful when the result is wrong. A killed hypothesis with a pre-regged SHA256 is evidence. Without one, it's a story. Your "boring evidence that ages better" framing is going in the paper.
That is why I liked the pre-registration angle. It makes the result harder to narrate away afterward. Even a killed hypothesis becomes useful because it protects the original question from being rewritten into something easier.
The limitation you flagged in passing is the most interesting thing in the post: pre-registration via git commit, and you could amend the commit.
That isn't a small caveat, it's the whole mechanism. Pre-registration buys nothing from the writing-down. It buys from the timestamp, and a timestamp is worth exactly what the clock is worth. Git commit dates are author-controlled. GIT_COMMITTER_DATE will set them to anything you like, so the commit on its own proves nothing to a skeptic who doesn't already trust you.
Cheapest upgrade for a solo setup, no registry needed: make the run witness the commitment. Hash the pre-registration (sha256 over the script header, the conditions, and the scoring regexes), then write that hash into every trial record you send. The provider timestamps those. You can still amend the commit afterward, but you can't amend 600 API records held by someone who isn't you, and the hash stops matching. Anyone can recheck it in a minute.
If you want a public clock instead, post the sha256 as a comment here before the run. Forem's timestamp isn't yours to rewrite. Ugly, free, works.
The other half worth separating: pre-registration protects two different things and you did the harder one. Predicting the direction is the part everyone talks about. Committing the scoring rule before seeing data is where p-hacking actually lives, and your regexes were frozen before execution. That's what makes the null credible, more than the prediction being written down.
And the null being reported at all is the point. A wrong prediction with a real timestamp is evidence. A right prediction with no timestamp is a story.
Dipankar, thank you — this comment changed what got built.
You said "hash the pre-registration and write that hash into every trial record." Done.
pre_register.pynow hashes the hypothesis + conditions + scoring regexes →b9ef83f7f890efe861e8b6b789f9fdbf. Re-extracting from the script produces the same hash. Deterministic. Third-party verifiable.You also separated what pre-registration protects — I hadn't made that distinction before you named it. Committing the scoring rule before seeing data is where p-hacking lives, and the regexes were frozen before execution. The prediction direction was the part I was focused on. You're right that the scoring freeze is the load-bearing one.
On the Forem comment as public clock: I'll do that for the next run. Post the hash here before the API calls start. Ugly, free, works — exactly the right design constraint.
The SHA256 scheme is at
github.com/YuhaoLin2005/paper-validator→pre_register.py. Full write-up with the hash verification in the community experiments doc. Thanks for the push — it turned a caveat into a mechanism.