When an AI analyst tries 20 things, pure noise can look like a discovery (and 3 habits that may help)

Disclosure: I drafted this article with the help of an AI assistant (Claude) and edited it myself. Every number below comes from code in the linked repository, so you can rerun it.

The short version

If you hand a dataset to an LLM and say "analyze this", there may be a built-in trap: it can try many analyses without getting tired, and the one that looks best may be the one that gets reported. Nothing has to be dishonest for this to go wrong. If the report does not say how many things were tried, a chance wobble can read like a finding.

You may not need statistics training to guard against it. Three habits, all doable inside a chat, could reduce much of the risk:

1. Analysing data you already have: split it in three, and keep the last part away from the AI.
2. Comparing options (prompts, models, settings): have the AI write down what will be compared before it looks, and freeze that.
3. Predicting something: save the prediction with a date, and wait for the outcome.

What actually happens (a short simulation)

Make data where nothing is going on: two versions of a system are identical, and every "metric" is random noise. Now act like an analyst who measures 20 different ways and reports the best one.

I simulated 2,000 such studies (200 items each, 20 metrics each):

What the analyst does	Studies that look "significant"
Names one metric in advance	4.8% (the designed 5%)
Tries 20, reports the best	66.3%
Tries 20, but accounts for having tried 20	6.0%

The textbook figure for "at least one of 20 independent tests comes out below 5% by chance" is 1 − 0.95²⁰ ≈ 64%, and the simulation is close to it. In this simulation, with no real effect at all, about two out of three studies produced a "finding". Real analyses try variants that resemble each other, so the real number will differ.

This pattern is known as p-hacking, data dredging, or the multiple comparisons problem.

You can reproduce it with the standard library (Python 3.12, random.Random(1)):

import math, random
from statistics import NormalDist, mean, stdev

N = NormalDist()

def p_value(diffs):                       # two-sided z-test on per-item differences
    se = stdev(diffs) / math.sqrt(len(diffs))
    return 2 * (1 - N.cdf(abs(mean(diffs) / se)))

rng = random.Random(1)
first = best = 0
for _ in range(2000):                      # 2,000 studies, no real effect anywhere
    ps = [p_value([rng.gauss(0, 1) for _ in range(200)]) for _ in range(20)]   # 20 metrics, 200 items
    first += ps[0] < 0.05                  # one metric named in advance
    best += min(ps) < 0.05                 # best of 20, reported as if it were the only one
print(first / 2000, best / 2000)           # 0.048 0.663

This may not be only a thought experiment. Several recent preprints (not yet peer reviewed) report that AI coding agents can drift in a similar way:

• The same data with different analysis choices can lead to different conclusions (arXiv 2602.18710).
• Blunt requests for a particular result were refused in one study, but softer wording such as "try alternative approaches" got through (summary of Asher et al.).
• The final report may hide how much searching happened (The Agentic Garden of Forking Paths).

Common traps

Trap	What can go wrong	Habit
Try many things, report the good one	Chance can look like a finding	1, 2
Choose the metric or threshold after seeing results	The conclusion may be tailored to the data	2
Draw conclusions from too few items	It may not replicate	1, 2
"Predict" something whose outcome is already known	It can seem to work every time	3
Read "no significant difference" as "the same"	You may just not have been able to see it	2
Trust an LLM's scoring without checking it against people	The scorer's quirks can leak into your result	2
Mistake a confident tone for correctness	You may trust numbers that were never computed	see "What this does not stop"

Habit 1: analysing data you already have

Split it three ways. Keep the last part away from the AI.

Part	Use
Explore	Look at anything. What you find here are ideas, not conclusions.
Validate	Compare the ideas and pick one.
Confirm	Data nobody has seen. One look. Only this part supports a conclusion.

First check there is enough data. Splitting may be pointless if the final part is too small to show anything. This one-page calculator (offline, English and Japanese) tells you what share of the data the final test would need: plan-calculator.html. If you do not have enough, the honest move is to collect new data going forward and make that the final part.

A prompt for the explore phase:

The attached file is the EXPLORATION set. A separate confirmation set exists and you will not see it.
Work in this order:
1. Check the columns and missing values and note anything odd.
2. Number every analysis you try and keep all of them, good and bad.
3. End with: how many analyses you tried, and up to 3 promising ideas. Do not state conclusions.
Run code for every calculation and paste the output. No mental arithmetic, no guessed numbers.

A prompt to freeze the plan before the confirmation run:

From the exploration, decide what we will test on the confirmation set. Write down:
- the single claim to check
- the metric and its direction (higher or lower is better)
- how many items we will use
- what counts as "a difference"
I am saving this text now. It must not change after we see the confirmation data.

And the confirmation run:

This is the confirmation data. Evaluate it once, exactly as frozen above.
If the result is disappointing, do not change the metric or the threshold; report it as is.
Label any extra analysis "exploratory" and keep it separate from the conclusion.

Habit 2: comparing options

Write down what is compared and how, before looking.

If you look, tweak and look again before the planned amount of data has arrived, no version of your approach may ever collect enough items to support a conclusion (top). Fixing the plan first, running one approach to the planned size, and treating early looks as hints keeps the verdict for the end (bottom).

We will compare prompt A and prompt B. Before touching the data, decide:
- the metric(s) that will decide the question (at most 2) and their direction
- metrics we will look at only for ideas (not for conclusions)
- how many items we will use
- how many variants we plan to try in total
I will save this. If you change the plan later, say so explicitly and keep the original.

Three things matter most:

• Keep the deciding metrics few. Every extra one adds chance.
• Count what you tried. If you tried 20 variants, read the best one as "the best of 20".
• "No difference" is not "the same". If the claim is "the cheaper option is not worse", agree on how much worse is acceptable first.

Habit 3: predictions

Save the prediction with a date, then wait. This is probably the cleanest habit because the outcome does not exist yet: nobody, including the AI, can peek.

Give the probability (0 to 1) that the following happens, with at most 3 lines of reasoning.
Event: <write it here>
Outcome known on: <date>
I will store this answer with today's date and review it once the outcome is known.

After enough of them, count: of the things you called "70%", did about 7 in 10 happen? That is calibration, and it shows how over- or under-confident the model really is.

What this does not stop

• Numbers the AI never computed. Always require "run code and paste the output."
• Accidentally showing the confirmation data. Keep it in a separate file and do not attach it to the chat.
• Concluding from too little data. If there is not enough, "no conclusion possible" is the honest report.

Optional: keep the record in a ledger

The habits above work in a plain chat. If you also want the plan and the records to be checkable afterwards (or want an agent to do the bookkeeping), I wrote a small open-source tool, forecastseal (standard library only, Python 3.10+, local, no network):

• the plan (metrics, planned sample size, alpha, optional non-inferiority margin, the list of evaluation items) is sealed before any result;
• records go into an append-only JSONL ledger where each entry includes the previous entry's hash, so edits, deletions and reordering are detected by fseal verify;
• comparisons are paired bootstrap intervals; looking before the planned sample size needs --interim, is logged, and gives no verdict;
• one self-contained HTML report.

Here is the "20 metrics" simulation from above in a ledger (examples/noise_demo.py). Only metric-01 was declared as the deciding metric before the data; the other 19 are exploratory:

Unadjusted best metric: metric-17 (p = 0.0070).
  metric-01: primary (declared before the data); interval [-0.370, +0.455] -> includes 0
  metric-17: exploratory (found after looking); interval [-0.842, -0.119] -> excludes 0
             but labelled exploratory: not a conclusion
  Every look is in the ledger: 2 analyses logged; verify: intact

The tool does not make a weak analysis strong. It tries to make it harder to bend a result without leaving a trace.

Limits worth knowing: timestamps come from the local clock (the chain only shows nothing changed after you anchor the head hash somewhere you do not control); deleting the last entries is detectable only if you saved an earlier head hash; and for the statistics themselves, libraries such as evalsig or errorbars are richer than this.

Repository: github.com/as-we/forecastseal

Summary

• One common failure in AI-assisted analysis may be chance turning into a discovery.
• Three habits could help reduce it: split the data and hide the last part, freeze what you compare before looking, let predictions wait for outcomes.
• Check whether you have enough data before you start. If not, collect it going forward.