This is a submission for the Kaggle Benchmarking Challenge
I built an AI benchmark where checking your own decision has a cost.
Then I tried three increasingly strong ways to make the system challenge itself:
- tell it to seek counterevidence,
- give it an executable way to test hypothetical consequences,
- automatically put those consequences into its decision path.
None produced a reliable average improvement in the final decision.
That was not the result I expected.
And it became the most interesting part of the benchmark.
What I Benchmarked
Decision-Conditioned Falsification asks a model to act in a small world with three possible hidden mechanisms.
The model knows the possible mechanisms, but not which one is active.
It has a choice:
ACT now
or
TEST → pay a cost → observe evidence → decide
Sometimes testing is the right move.
Sometimes the exact same desire for more information is a waste of points.
That distinction is the benchmark.
It does not reward how much the model doubts itself, how many tests it runs, or how often it changes its mind.
It rewards:
net operational utility.
A simple way to think about it:
If checking costs 5 points, is knowing more actually worth 5 points before you act?
That also lets the benchmark separate three things that are often collapsed together:
- Information gain: how much did I learn?
- Falsification: did the evidence contradict my hypothesis?
- Decision value: did learning it actually help me make a better decision?
They are not the same thing.
The environment contains three deterministic hidden mechanisms, W0, W1 and W2. The agent can compose experiments from primitive operations. Different sequences distinguish different subsets of the possible mechanisms, and there is no single cheap experiment that identifies everything. The public benchmark contains six decision configurations × three hidden worlds = 18 isolated episodes per model. Experiment costs, priors and operational rewards vary across configurations. This creates cases where: The primary score is prior-weighted net operational utility, averaged across the six configurations. An exact solver calculates optimal values only for post-decision diagnostics. It never tells the evaluated model which world is active, which experiment to run, or which action is optimal.Click to see how the benchmark works
Models Tested
The public Kaggle benchmark currently contains four model runs:
- GPT-5.4 nano
- Gemini 3.5 Flash
- Claude Haiku 4.5
- Gemini 3.7 Flash
Gemini 3.7 Flash was used during Kaggle task creation, so I preserved the run rather than silently removing it.
The other three were selected before the final evaluation to give the benchmark several model families instead of many variants from one provider.
The live leaderboard is available on Kaggle.
But the most interesting result came from using the environment to test the architecture around the model.
Findings
Here is the entire experimental story in one table:
| What I changed | What happened |
|---|---|
| Tell the model to seek counterevidence | No robust average benefit: −1.45 pts |
| Give it executable hypothesis prediction | It used the capability only 1 time in 360 assisted episodes |
| Automatically inject computed consequences before action | Still no demonstrated average benefit: −0.39 pts |
That progression changed how I think about AI self-checking.
1. Asking for counterevidence was not enough
The first comparison held the model, environment, tools, budgets and output structure constant.
One condition simply added an explicit instruction to consider evidence that could contradict assumptions supporting its plan.
The average difference was:
−1.45 points
95% interval: [−2.93, +0.01]
So explicitly asking for falsification did not show a robust aggregate benefit.
The traces suggested another problem: the model sometimes declared predicted outcomes that were physically impossible under every mechanism in the environment.
Instead of making the prompt more forceful, I changed the system.
2. Making hypotheses executable was not enough either
I added a deterministic function that could answer:
If W1 were true, what would this experiment produce?
This was kept separate from actually observing the hidden world.
predict(hypothesis, experiment)
≠
run_experiment(experiment)
The first calculates a hypothetical consequence.
The second produces real evidence.
The surprise was simple:
the model almost never used the prediction capability.
It was invoked once across 360 assisted episodes.
That gave me a much clearer engineering lesson:
A capability being available inside an agent does not mean that capability participates in its decisions.
So I removed the choice to invoke it.
3. Even automatically supplying the consequences was not enough
In the final experiment, the model first proposed a decision.
Before that proposal could become an action, the system automatically calculated its consequences under the possible mechanisms and put them directly into the model's context.
The control received the same opportunity to reconsider, but without those computed consequences.
This time the model could not simply forget to open the tool.
The projection reached the model in 170 of 180 applicable first reviews.
The result:
Integrated consequences − Neutral review: −0.39 points
95% interval: [−1.53, +0.80]
At the first review itself, where both branches started from exactly the same provisional decision, the difference in decision value was just:
−0.088 points
95% interval: [−0.454, +0.402]
The intervals do not prove equivalence.
But I still did not observe an average improvement from inserting the correct hypothetical consequences into the decision path.
A deterministic PREDICT function could calculate consequences under public mechanism rules without observing the real hidden world. Across 360 assisted episodes it was called only once. The prespecified factorial interaction for offering the capability was: −2.71 95% interval: [−5.33, −0.60] Because the capability was almost never invoked, this estimates the effect of offering that tool/protocol, not the effect of systematic executable prediction. The integrated projection was delivered in 170/180 applicable first reviews. The three studies used different protocols and are not pooled into one effect estimate.Click to see the exact controlled-study details
Explicit falsification
Optional executable hypotheses
Integrated consequences
The result I actually care about
It is tempting to think of this as one chain:
The system can check something
↓
The check is correct
↓
The check reaches the model
↓
The model uses it
↓
The decision improves
My experiments made those steps come apart.
A check can be correct.
It can reach the model.
The model can reconsider.
And the final decision can still fail to improve.
That does not mean self-falsification is useless.
It means:
Correct epistemic machinery is not automatically valuable decision machinery.
A safeguard should not be considered successful simply because the safeguard itself works.
It has to improve what the system eventually does.
The harder problem was deciding when to check
Another thing surprised me.
The difficult part was often not finding a counterexample.
It was knowing whether another experiment was worth running at all.
Models sometimes stopped when another test still had positive decision value.
They also sometimes paid for information when acting immediately was already better.
An experiment can reveal a lot and still be useless for the current decision.
A hypothesis can be falsified without changing the best action.
And a model can obtain useful evidence and still choose the wrong action.
That gap now interests me more than simply adding another verification layer.
How does evidence become causally connected to action inside an AI-native system?
This is deliberately a small, controlled benchmark. I wanted a benchmark capable of telling me that an architecture I found appealing did not produce the improvement I expected. Otherwise it would not have been much of a falsification experiment.Click to see the main limitations
My Benchmark
Decision-Conditioned Falsification
👉 View the benchmark on Kaggle
The benchmark is published under Apache 2.0.
One display note: Kaggle currently renders the raw task value in the leaderboard table using percentage-style scaling, while the benchmark summary displays the underlying value correctly. I kept the frozen task unchanged and use the underlying score in the benchmark documentation.
I started with this question:
Can an AI system prove its own decision wrong when checking has a cost?
After building the benchmark, I would ask a different one:
Can we design the path from hypothesis, to evidence, to action so that discovering you are wrong actually improves what you do next?
That became the more interesting benchmark.
Top comments (0)