Yesterday I froze the rules first; today the results of the label-redesign experiment came in. Zero passes, so I kept the current setup, and an outside AI review corrected my reading from "it doesn't make money" to "the data can't tell whether it makes money."
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
The rules I froze yesterday, and the results that followed
Yesterday I nailed the pass bar, the window, and the seeds into a pre-registration document before looking at a single number.
Today I actually ran that experiment and read it out. The first candidate was "align what the model learns to how I actually trade (buy next morning, hold a few days)," plus two variants that narrow the training universe.
The verdict can only be one of three values: pass, reject, or can't-tell. The result was zero passes.
So, exactly as I decided yesterday, I changed nothing and kept the current setup.
One of the three "improved the ranking, but..."
One of the three variants — the one trained on the more liquid names — clearly improved the prediction ranking itself.
It pointed the same way across the whole six-year window, and it held up under the secondary measures too. On the ability to rank stocks alone, it was a real improvement.
But the pass bar I'd set in advance demands both "statistically significant" and "consistent quarter to quarter."
This candidate cleared the first condition but fell short on quarter-to-quarter consistency. By the rules, it can't be adopted.
Not "it doesn't make money" but "I couldn't measure it"
There's a natural reading here: "the ranking got better, but it didn't translate into money."
Right after the readout I had an outside AI review the result, and it overturned exactly that reading.
The crux was statistical power. The smallest profit effect this experiment could detect was larger than the entire edge this strategy currently has.
In other words, the design simply couldn't distinguish whether profit improved or not from these six years of data. The right description isn't "no effect" but "can't tell whether there's an effect."
Breaking it down, the cost savings from lower turnover were distinctly positive, while the change in gross return was within noise. The two overlapped, so the net effect only looked like it sat near zero.
The owner's idea was rejected, but that's not a disproof
The variant that narrowed the training universe further to the larger-cap names came back as a reject. It got significantly worse in the negative direction.
But this one shouldn't be read straight as "the idea is wrong" either.
Narrowing the universe that much left too few samples to train on each day, and most blocks early-stopped before the model could really learn.
So "the effect of shrinking the sample" and "the effect of plain underfitting" don't separate. More precisely, it's "rejected at this setting and this sample size," not a disproof of the idea itself.
No post-hoc reinterpretation, and power feeds the next design
The outside AI's conclusion was matter-of-fact: no defects in the implementation or the verdict, applying this result is not recommended, and there's no legitimate way to re-measure it from the same six years.
So I pinned down two follow-ups. One is that I won't later reinterpret the "ranking improved" observation as grounds for adoption without new evidence.
The other is to feed the power I actually measured this time into the pre-registration design for the next candidates. The point is to design experiments already knowing what effect size they can detect.
I also scheduled a forward sign-only check for next year.
Does re-scoring give the same score?
I ran a separate experiment today too: checking whether re-running the scoring process itself yields the same score.
Re-scoring under the same settings doesn't give identical results — it gives similar-but-noisy values.
Pooling three nights, the readout landed on a borderline gray zone: neither pass nor reject.
At the border, running it once more doesn't settle anything. So I decided to pool two more nights and re-judge exactly once on a five-night average, and I kicked off the fourth run tonight on the GPU.
In effect, I'm measuring not the value but the noise floor of the measurement itself.
Why this matters
Both threads today came down to the same question: is the difference I see a real improvement, or just noise?
The hardest habit to internalize is telling "it doesn't work" apart from "I can't tell." Both land on "keep current," but the former closes the door while the latter hasn't even opened it yet.
When you see the ranking improve, the temptation to treat it as evidence of better profit is always strong.
There was only one way to beat that temptation: freeze the pass bar before looking at the numbers, and leave what you can't detect as "don't know."
Top comments (0)