A stranger on DEV.to said "run this experiment." I ran it at n=600. Here's what happened — including the part where he caught me reporting post-hoc findings as if they were planned.
The Backstory
Mike Czerwinski read my article about AI agents following rules. He proposed a specific experiment: test prose-format rules under a mechanical gate. Can you get both 100% compliance AND deep reasoning by writing rules as narrative instead of commands?
I had pilot data suggesting yes. Mike pointed out the pilot ceiling was probably noise. He was right. I designed a full 2×2 factorial experiment and ran it at n=600.
But I did something else first. Something I'd never done before.
The Pre-Registration (Solo-Researcher Edition)
I don't have an OSF registry. I don't have an advisor. What I have is a Git repository and a Python script.
Before running a single API call, I wrote the hypothesis into the experiment script header:
"""Pre-registered hypothesis: Format effect on reasoning depth
is LARGER under GateGuard-OFF."""
The script defined all 5 rules, all 4 conditions, the deterministic regex scoring, and the analysis plan — committed BEFORE execution. git log shows the timestamp. That's my pre-registration: a timestamped, immutable snapshot of what I predicted before I saw the data.
It's not peer-reviewed. It's not a third-party registry. But it's honest, and it creates a paper trail that can't be rewritten after the fact.
The Experiment
Design: 2×2 factorial — Format (code / prose) × Gate (ON / OFF)
Rules: 5 governance rules (delivery gate, health check, self-review, fact-check, self-model regeneration), each in both code and prose format
Trials: 30 per condition per rule = 600 API calls
Model: DeepSeek V4 Pro, temperature=0
Scoring: Deterministic regex — no LLM judge
| Condition | Compliance | Reasoning ± SD |
|---|---|---|
| Prose + Gate ON | 91.3% | 3.23 ± 0.64 |
| Code + Gate ON | 99.3% | 2.82 ± 0.70 |
| Prose + Gate OFF | 90.0% | 2.92 ± 0.94 |
| Code + Gate OFF | 98.0% | 2.67 ± 0.71 |
The Pre-Registered Hypothesis: KILLED
My pre-registered prediction was that format would matter MORE when the gate was off — that code format and mechanical enforcement overlap, so removing the gate would reveal format's true effect.
The data said no:
- Gate ON: d(code−prose) = −0.277
- Gate OFF: d(code−prose) = −0.250
The format effect on reasoning is nearly identical regardless of gate status. The hypothesis was wrong.
This is the point of pre-registration. If I hadn't written down the prediction beforehand, I could have looked at these numbers and said "I predicted this." Post-hoc rationalization is cheap. A timestamped Git commit isn't.
What the Data Said Instead (Three Things)
1. The ceiling was noise. My pilot found code_OFF reasoning = 4.42. At n=30, it's 2.67 — below ALL gate conditions. Mike's skepticism was correct. Small-n pilot ceilings are not findings.
2. Gates improve reasoning, not suppress it. The gate added +0.32 reasoning in prose and +0.15 in code. Everyone's intuition is that enforcement constrains thinking. The data shows the opposite: mechanical structure improves reasoning depth in both formats. The gate acts as cognitive scaffolding, not a straitjacket.
3. Prose + Gate is the best configuration for reasoning depth. Prose format consistently outperforms code format on reasoning (~0.25 SD advantage), regardless of gate status. Combined with the gate's structural boost, prose+gate produces the deepest reasoning (3.23 vs 2.82 for code+gate). Cohen's d = 0.605.
The practical takeaway: if you care about compliance, use code format + gate (99.3%). If you care about reasoning depth, use prose format + gate (3.23/5).
The Part Where I Got Caught
I also reported a per-rule breakdown: prose helps meta-cognitive rules (self_review: +1.08 over code) but hurts precision-dependent rules (fact_check: −0.21). I suggested "hybrid deployment" — prose for some rules, code for others.
Mike replied with a methodological question: "Was the per-rule breakdown pre-registered?"
It wasn't.
The data was always going to be collected — rule_id is in every trial, all 5 rules were defined before execution. But the specific pattern I reported was discovered after seeing the results, not predicted beforehand. Pre-registered per-rule predictions would be strong evidence. Post-hoc pattern finding is exactly the kind of result that regresses on the next run.
I updated the README to flag this. The hybrid deployment recommendation now rests on the pre-registered overall effect (d=0.605), with per-rule heterogeneity marked as exploratory. Mike also pointed out that the 8% regex detection gap in prose+gate cases means the fact_check measurement might be an artifact, not a real decline.
This exchange — someone catching a methodological gap in your work, and you fixing it publicly — is what peer review is supposed to be. DEV.to comments aren't peer review. But they're also not nothing.
What Pre-Registration Buys (Even Without an Advisor)
I'm an undergraduate with one laptop and no lab. Pre-registration for me looks different than for a funded research group. But the core mechanism is the same:
- Write down what you predict. Script header, design doc, Git commit message — any timestamped, immutable record.
- Define your scoring before you see data. My regex patterns were committed before execution. No tuning after seeing results.
- Report the null result. The pre-registered hypothesis was wrong. That makes the finding MORE credible, not less — because I can't have p-hacked my way to NOT_CONFIRMED.
- Separate pre-registered from exploratory. When someone asks "was that planned?", have an honest answer.
The point isn't to predict correctly. It's to make "wrong" useful. A null result with a timestamped prediction is evidence. A null result with post-hoc explanation is a story.
Honest Limitations
Single model (DeepSeek V4 Pro). Single rater (deterministic regex — consistent but limited; the 8% detection gap means some effects are measurement artifacts). No holdout sample. Per-rule breakdown is exploratory. Pre-registration was via Git commit, not a public registry — I could theoretically amend the commit (though the GitHub timestamp trail would show it).
I'm working on a pre-registered per-rule replication with second-rater scoring for the fact_check rule specifically. If you have suggestions for a lightweight pre-registration workflow for solo researchers, I'm listening.
👋 林宇浩 — Building verification infrastructure for AI agents. One laptop, 50+ sessions, 1,200+ API calls. github.com/YuhaoLin2005/hermes-workspace
Top comments (1)
The limitation you flagged in passing is the most interesting thing in the post: pre-registration via git commit, and you could amend the commit.
That isn't a small caveat, it's the whole mechanism. Pre-registration buys nothing from the writing-down. It buys from the timestamp, and a timestamp is worth exactly what the clock is worth. Git commit dates are author-controlled. GIT_COMMITTER_DATE will set them to anything you like, so the commit on its own proves nothing to a skeptic who doesn't already trust you.
Cheapest upgrade for a solo setup, no registry needed: make the run witness the commitment. Hash the pre-registration (sha256 over the script header, the conditions, and the scoring regexes), then write that hash into every trial record you send. The provider timestamps those. You can still amend the commit afterward, but you can't amend 600 API records held by someone who isn't you, and the hash stops matching. Anyone can recheck it in a minute.
If you want a public clock instead, post the sha256 as a comment here before the run. Forem's timestamp isn't yours to rewrite. Ugly, free, works.
The other half worth separating: pre-registration protects two different things and you did the harder one. Predicting the direction is the part everyone talks about. Committing the scoring rule before seeing data is where p-hacking actually lives, and your regexes were frozen before execution. That's what makes the null credible, more than the prediction being written down.
And the null being reported at all is the point. A wrong prediction with a real timestamp is evidence. A right prediction with no timestamp is a story.