DEV Community

YuhaoLin2005
YuhaoLin2005

Posted on

My Experiment Showed Zero Effect. A Statistician Told Me My Measurement Was Broken.

#research #datascience #python #ai

Last week, I ran an experiment that failed.

The hypothesis was simple: syllogistic prompts ("Major premise → Minor premise → Therefore...") should make AI models internalize rules more deeply than imperative prompts ("You MUST..."). I designed 8 probes, ran them across 3 conditions, and...

Cohen's d = −0.148. Direction: ~50%. Bayes Factor: < 1 (supporting the null hypothesis).

Zero effect. Nothing. I was ready to scrap the whole idea.

Then three experts looked at my data and said the same thing: "Your measurement tool is broken."

The Problem Hiding in Plain Sight

Here's how I was measuring "constraint internalization":

  1. Give the model a binary choice (A = compliant action, B = violating action)
  2. Ask it to pick A or B
  3. Compare the log-probability of token "A" vs token "B"
  4. Differential = logprob(A) − logprob(B)

Seems straightforward. But DeepSeek's API has a quirk: it only returns the top-20 logprobs. If your comparison token isn't in the top 20, you get nothing. My code assigned −10.0 as a sentinel value for missing tokens.

Here's what that does to your data:

# What I thought I was measuring:
#   Format effect = Syllogistic(A-B) − Imperative(A-B)
#   e.g., (+5.2) − (+4.8) = +0.4

# What I was actually measuring:
#   Syllogistic: B-token NOT in top-20 → gets -10.0 sentinel
#   Imperative:  B-token IN top-20 → gets -0.8
#   "Format effect" = huge number made of noise
Enter fullscreen mode Exit fullscreen mode

4 out of my 8 probes had this artifact. The "large effects" I was excited about in the exploratory phase? Garbage. The violating token simply wasn't in the API's returned top-20, and my sentinel value fabricated a massive logprob gap.

This is what the statistics expert on my review panel called "garbage in, garbage out."

The Fix: Pre-Validate Every Probe

The solution is obvious in retrospect — and that's what makes it a good lesson:

Before running the experiment, verify that your measurement tool actually works.

I built probe_validator.py: for each of 40 probes, run it in all 3 conditions (baseline, imperative, syllogistic), and check:

  1. Does token "A" appear in the top-20 logprobs?
  2. Does token "B" appear in the top-20 logprobs?
  3. Does the model actually choose A or B?

If any check fails → drop the probe. Only run the experiment with probes that pass all three gates.

I also redesigned the probes with a critical formatting fix. The original probes ended with "我应该选：" ("I should choose:") — which caused the model to output "选" (choose), "我" (I), or "根据" (based on) instead of A or B. The new probes all end with "A 或 B？" ("A or B?") — forcing the model to commit to a token choice.

What Happened When I Re-Ran

40 validated probes. 3 conditions. 120 API calls. Total cost: ~$0.60.

Metric Pilot (n=8, broken) Confirmed (n=40, validated)
Cohen's d_z −0.148 +0.578
Bayes Factor (BF₁₀) < 1 282,399
Bootstrap 95% CI crosses zero [+3.39, +11.17]
Direction ~50% 80% (32/40)
Leave-one-out t range unstable [3.43, 4.89]

The effect was real all along. I just couldn't see it through the noise.

Cohen's d = 0.578 is a medium-to-large effect. BF₁₀ = 282,399 means the data is 282,000 times more likely under the alternative hypothesis than the null. The bootstrap confidence interval doesn't cross zero. Leave-one-out analysis confirms no single probe is driving the result.

And here's the secondary finding: the format effect doesn't depend on constraint type. I tested 4 categories (action, epistemic, structural, meta), 10 probes each. ANOVA: F(3,36) = 0.26, η² = 0.02 — not significant. Syllogistic prompts help across the board.

What This Actually Means

The syllogistic format doesn't just make rules sound more authoritative. It changes how the model internally weights constraint-relevant tokens. "You must check X before Y" gets processed as an instruction. "Premise: X must be checked before Y. This action involves Y. Therefore, check X first." gets processed as a logical chain.

This converges with independent research: Pender (2026, Zenodo) showed that prompt format changes attention routing patterns in transformer models.

But here's what I'm not claiming: that syllogistic prompts are a magic fix. When I ran a separate 150-task compliance experiment with active mechanical enforcement hooks, compliance hit 99.3% with both formats. Format affects internal processing, but mechanical enforcement dominates behavioral output.

The Meta-Lesson

I spent the first iteration running t-tests and computing Cohen's d. None of that mattered because my measurement was broken.

Three things that actually moved the project forward:

  1. Show your raw data to someone who knows statistics. The expert panel spotted the floor artifact in 5 minutes.
  2. Check your tools before your hypotheses. The probe validator took 30 minutes. It saved me from publishing garbage.
  3. Report the failed pilot. d = −0.148 → d = +0.578 is a better story than just the final number.

Code, Data, and Reproducibility

Everything is open source at github.com/YuhaoLin2005/hermes-workspace: 40-probe pool, pre-experiment validator, two-experiment architecture with bootstrap CI + Bayes factor + leave-one-out. Full JSON results. ~$0.60 in API costs.

I'm an undergraduate at Fujian Agriculture and Forestry University researching how AI agents internalize behavioral constraints. Single model (DeepSeek V4 Pro). No institutional funding. No advisor. One person, one laptop — working on it anyway.

Top comments (0)