A wiring mistake broke the overnight batch; recovering from it also exposed a bypass loophole and a two-day-old test-contamination bug
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
Ten hours, and the batch never once finished
Yesterday, while wiring up a new intraday observation pipeline, one line inside the model-selection function got accidentally moved under a conditional.
The problem was that the same function is also used by the regular overnight production batch. On that path, the condition never triggers, so a variable was left undefined and the batch crashed.
From 8pm to 6am, it retried automatically every five minutes and died at the exact same spot every time. Close to a hundred attempts over ten hours, and not one of them finished.
There was no self-alerting mechanism, so it only surfaced the next morning when I directly asked "was everything okay overnight?" I reverted the one broken line immediately and restarted all resident processes.
I also checked why over a thousand existing regression tests missed this. It turned out every test that touched this function replaced the whole thing with a mock, so none of them ever actually exercised the real internal logic.
So I added a new test that actually runs the default code path. I confirmed it failed with the exact same error against the pre-fix code before committing it as passing.
Recovery surfaced a loophole in the bypass path
A batch that never finished overnight meant the trading model's ranking was now two days stale by morning.
I decided to rerun the heavy model from scratch and let it finish by afternoon, relying in the meantime on the existing safety gate that filters out daytime automated trading rounds.
Once the model finished in the afternoon, I ran a script that re-froze the day's trading criteria from that output, then triggered a trading round manually.
That manual trigger failed completely — no orders went through. The cause: a flag meant only for scheduled runs was hardcoded into the execution unit itself, so even the manual run got misclassified as "a scheduled round that arrived too late" and was blocked.
I found a workaround (running one layer lower, bypassing the unit) and triggered it successfully. I left the loophole itself as a follow-up item for a proper fix later.
The live account had a similar incident
I briefly stopped a few automated trading timers during the day. One of them was quietly restarted by a separate auto-recovery mechanism I hadn't accounted for.
As a result, the live account executed a handful of trades against criteria that were still frozen from two days earlier. I asked an AI model to verify safety, and it confirmed that the order execution and safety layer(new tab) — the turnover cap and the loss ceiling — had both worked correctly, and that the worst-case loss was very small.
Based on that verdict, I decided to leave the trades that had already gone out as is.
While investigating this, I also turned up a second risk of the same kind: a recovery trigger I'd registered earlier that morning turned out to be another mechanism capable of reviving the same timer.
Turning it off right away would have also killed the afternoon's automated trading restart, so I had to make a direct call: keep it, since I wanted the afternoon restart to go ahead.
The lesson: whenever something gets stopped temporarily, first check whether any other automation could bring it back, and handle all of them together.
A two-day-old ledger contamination bug, finally traced to its root
Yesterday, a paper-trading account showed unexplained ghost trades, and the exact cause couldn't be pinned down at the time.
Today the same symptom reappeared, and this time tracing it backward from the recurrence found the root cause: a regression test file.
That test exercised the real code path of the internal function that records fills, without mocking it out. So every time the full test suite ran, it quietly wrote a handful of fake trades into the live operational database.
I backed up the contaminated rows, precisely identified and deleted them, then reconciled the ledger against actual holdings.
I also fixed the test to properly isolate that function. As a general defense against this class of bug recurring, I reconfirmed the rule that any new database file used in production gets registered in the test-isolation list right away.
The same day also produced a wave of test failures from an unrelated full suite run during market hours — I confirmed those were false positives, triggered by the test suite mistaking live production writes for corruption. I reaffirmed the rule that the full regression suite shouldn't run during market hours.
Top comments (0)