The experiment itself was supposed to decide whether a new prompt line helps — instead, the harness running it broke three different ways in one evening.
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
I ran an experiment tonight that had been in design for the past few days.
It's an A/B test to check whether adding one extra line of U.S. overnight market data to the AI's input actually improves its judgment, or just adds noise.
Same dates, same tickers, run through two different input conditions and compared. The plan was to only adopt the change if the evidence actually supported it.
Bug one: intersection, not union
The experiment freezes two calendar days of input, and each day has a slightly different set of tickers with confirmed data.
On the first run, a ticker that was missing from one of the two days slipped into the combined list, and the whole run got rejected.
The logic that built the ticker list included a ticker if it existed on either day. I fixed it to require both days — an intersection, not a union.
Bug two: a retry quietly reused stale output
After fixing the first bug, I reran it — and this time it finished suspiciously fast.
It turned out that retrying under the same run tag triggered a "resume an interrupted job" feature, which reused leftover output from the earlier attempt.
No model call actually happened, so this repetition received the exact same input as the one before it — which makes a comparison experiment meaningless.
I added a step that quarantines any stale output under the same tag before a new run starts. It was a collision between the resume feature and a repeated-experiment design that happened to share a tag.
Bug three: the model paraphrased the marker, and the duplicate check misfired
The third bug showed up at the verification stage.
The experiment checks response quality partly by counting how many times a specific marker phrase appears. This time, the model paraphrased that phrase, and the check flagged it as a duplicate.
The marker itself was vague enough that the model had room to reword it. Redefining the marker more precisely and re-checking fixed it.
All three bugs shared something: none of them showed up during the lightweight pre-run checks.
The tell for the second bug, in hindsight, was that the first run finished faster than expected.
Smaller fixes on the side
The same day, I recalibrated the baseline used to daily-check numeric responses.
Looking at the last two weeks of data, the average had dropped noticeably from before. The old rule — "flag it if a specific metric hits zero" — was calibrated for a higher-average era, so at current levels it was raising a false alarm roughly once every five nights.
I changed the rule to only flag when two metrics hit zero at the same time.
A password-expiry warning also came up for the exchange login. I filed for a grace-period extension to buy time, then backfilled three days of derivatives data that had been missed in the meantime.
I also adjusted the time window that blocks a lightweight check process during the day. There's an evening job that computes next trading day's rankings after market close, and I shifted the rejection window boundary so the check process doesn't overlap with it.
Someone raised the question of whether that whole window should just move overnight instead, so I had a different AI review it — the conclusion was to keep the current structure as is.
Why this mattered
All three bugs today came from the harness running the experiment, not from the experiment's design itself.
The date-intersection logic, the collision between retry and resume, the ambiguity in the verification marker — all three were corners that hadn't been exercised before, simply because this was the first time running this kind of experiment.
None of them were catchable by lightweight pre-checks; they only surfaced once I actually ran it. That reinforced the habit of chasing down small anomalies — like a run finishing too fast — right away instead of letting them pass.
What's next
I didn't get through the full planned run volume tonight, so the remainder continues tomorrow evening in the same window.
Once all the results are in, I'll check whether the difference is statistically significant before deciding whether to actually adopt the change.
Top comments (0)