A hardware verdict flipped twice in one week, two verification tools turned out to be quietly missing errors, and a real brokerage account needed two manual human interventions
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
It's been a while since the last devlog, and it turned out to be the busiest week yet. We swapped the GPU entirely on Monday, and that verdict got reversed twice within a few days.
The same week, two newly built verification tools surfaced problems we'd been missing without noticing. And a real brokerage account needed two direct human interventions.
The GPU got swapped, and the verdict flipped twice
On Monday we replaced the graphics card with a different brand's model. The compute backend changed along with it — a big move.
Tuesday's production rollout, combined with tuning a separate speed-up technique, landed on "one new card does the work of two old ones." While we were at it, we ran a large code review on everything that had piled up recently and caught several serious issues, including a bug that could have allowed a double-buy.
Wednesday brought an unexpected self-inflicted error while investigating something unrelated (an old question about AI model size). A self-built groundedness checker turned out to be misjudging different formatting styles of the same value as errors. We invalidated it immediately and added a guard against the same failure mode recurring.
From Thursday night through Friday, we re-examined the evidence for buying yet another graphics card, and the verdict flipped twice in a single day. We ended up buying nothing, decided to return the expensive card already on order, and pivoted to answering the question with a small, near-zero-cost experiment instead.
Saturday (today) we were debugging that experiment's mid-run interruptions, while also exploring "using less GPU overall." We tested whether the morning report commentary could be produced with an AI subscription we already pay for instead of the local GPU — quality came out comparable or slightly better. A shared usage-cap issue with other tasks is still unresolved, so no final call yet.
Automated checks were quietly hiding gaps
We wired a new automated numeric checker into production reports. It immediately showed formatting errors in more than half of the tickers in every single daily report.
Looking back 45 days, the same class of error had shown up every day without exception. Before this tool existed, nobody knew. It was another case of "the system runs, but nothing is actually being checked."
A gauge that checks model reproducibility every day also died two nights running. The cause: another overnight job was holding the GPU, but the gauge decided "it's free" from a single instant read.
We switched it to directly check occupancy. Instead of spinning up a fresh instance, it now hands off from the overnight job right as that job wraps up, removing the failure point entirely.
A real account needed two human interventions
Background on order execution and safety-layer design(new tab) is in an earlier post. This week, those safety layers themselves caused two incidents.
On Thursday we found a specific position that had a sell order queued for days but kept not executing. The root cause was a structural flaw in the daily trading-budget allocation rule that always pushed small orders to the back of the queue.
While fixing the rule, one order that couldn't go through in time was handled manually — a real, irreversible trade on a live account.
Friday brought the opposite failure. A drawdown safety guard was reacting to momentary intraday dips, which incorrectly blocked two legitimate buy orders.
After review, the blocked buys were executed manually, and the guard was redesigned to react only to close-to-close comparisons instead of intraday snapshots.
During this, a separate bug surfaced in the tool that records manual trades into the ledger — it didn't have a "new buy" case defined, so the asset briefly showed up as if it were outside the ledger. We fixed it immediately by respecting the correct order (record the sell first, then the buy), and verification confirmed the ledger matched the live account exactly.
Also this week
- The documentation had grown too large, so we reorganized it into five purpose-based groups and added an always-on check for broken internal links.
- We created a document that collects the standing rules every session must check at startup, and wired it to load automatically.
- We wrote up the manual order procedure as a standalone document and linked it from several code paths that touch the live account, so it can't be missed.
Looking back, this was less a week of new features and more a week of re-checking things we thought were already working. The GPU verdict, the verification tools, and the safety guards each betrayed us at least once.
Fortunately we caught each one as it happened, and the live account balance matched the ledger exactly by the end of the week. Next week we should have results from the small experiment we redesigned this week.
Top comments (0)