A tie-break method for ranking failed a pre-registered evaluation bar, and I swapped it for a new one the same day.
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
A tie-break method gets rejected
The ranking logic that lines up stocks had a problem: when scores tied, the ranking could flip day to day. Frequent flipping meant unnecessary trades, and each of those trades cost fees.
A few days ago I introduced a tie-break method to reduce this and put it live. Before deploying it, I set a pre-registered bar: watch it for a fixed period, adopt it if the effect clears a threshold, reject it otherwise.
Today was judgment day, and the measured result fell well short of the bar. It also showed no statistically meaningful difference from the original state, so I confirmed the rejection.
Hearing that, I pushed back: "doesn't leaving it as-is just mean continuing to lose money — find a more sophisticated alternative." The current approach seemed to generate too many fees and lag behind the market in real time.
AI advisor, then an immediate swap
I asked an AI advisor to re-examine the root cause. The result was interesting. My own intuition — that the lag was the problem — turned out to be pointing the wrong direction. The real cause was that the raw score feeding the ranking itself swung almost randomly day to day.
In other words, the day-to-day ranking movement wasn't tracking any real trend — it was closer to noise. The advisor's suggestion: instead of using each day's raw score, average several recent days together to suppress that noise. That felt like a more fundamental fix than patching the tie-break mechanism itself.
I decided to rip out the old method and swap in the new one right away. Since speed mattered more to me than a longer validation window, I applied it to live trading the same day I verified it.
Before deploying, I ran a quick backtest on whatever historical data was available. I'd hoped for two months of history, but only about a month turned out to be usable.
Within that window, trading turnover dropped clearly, and cost reduction pointed the same direction consistently. Whether overall returns actually improved was harder to tell — the sample was too short. I decided to watch it for a few more days and judge that separately.
A whole day of code review
The same day, I also ran a large code review. I built a reusable review workflow that takes a model and a review intensity as parameters, then split everything changed since the last review into five parallel tracks and went through all of it.
I fixed every critical and important finding — fourteen in total — the same day, and also folded in nineteen lower-priority suggestions.
One bug the review turned up stood out. A live-account report was showing a return in the hundreds of percent. The cause: shares transferred between accounts were being excluded from the principal calculation but still counted in the valuation — the numerator was inflated while the denominator stayed flat.
I sent out a corrected report with the fixed numbers. A separate bug also surfaced where a test was firing real alerts into the production messaging channel — the mock only covered part of a test class, so I fixed the scope.
A root fix for false holdings-mismatch halts
I discovered that the broker's holdings-lookup call had a quiet failure mode: if one page of a paginated response failed, it would still return as if the whole call had succeeded.
That caused a recurring false mismatch — positions the account genuinely held would drop out of the lookup result — and each time, trading would halt. I changed the call to surface that partial failure as an explicit error instead of silently swallowing it, which eliminates this entire class of false halt.
Layers like this — order execution and the safety layer around it(new tab) — are, unlike the core stock-picking logic, something I'm comfortable documenting in more detail.
What's next
The new tie-break method still needs a few more days of watching for anomalies. After that, I'll judge separately whether it actually improved returns.
Top comments (0)