Speed really did improve, but I found out consistency was shaky — and in the middle of it all, a loose power cable knocked the machine into a reboot
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
The day isn't over yet, but here's where things stand so far. This one was very much a sequel to yesterday's story.
Yesterday's "flip", dug into further, turned out more complicated
Yesterday I wrote that pulling an old parallel-processing setup back out had made things noticeably faster. Digging deeper today, the picture turned out more complicated. The speed gain was real — I re-measured it and confirmed it.
But I also found that running the same input with the same settings repeatedly produced slightly different results each time. My first guess was interference between concurrent requests.
Setting up a control group ruled that out: running a single request repeatedly, with no concurrency at all, still produced slightly different results each time. The culprit wasn't concurrency — it was something more fundamental underneath.
It turned out to be a preexisting property: running the same computation on a GPU repeatedly doesn't reproduce the exact same floating-point operation order every time. Concurrency only amplified that wobble slightly; it wasn't the main cause.
So I decided not to use this parallel setup in the pipeline that actually does scoring, at least for now. If results wobble, there's no way to tell whether a difference between yesterday's judgment and today's reflects a real market change or just computational noise.
Instead, I pivoted to using the newly added second GPU to run two fully independent computations side by side — gaining speed without the wobble. It was a good example of how "a conclusion reached yesterday" can get flipped again within a day or two.
An overnight power cable accident, and the bug the reboot surfaced
Overnight there was also a purely physical accident — a loose GPU power cable caused the machine to reboot unexpectedly. Fortunately the services meant to run in the background came back up on their own after the reboot, and the database was intact with no damage.
Right after the reboot, though, I found a subtle bug. In the brief window after boot before the network had reconnected, yesterday's new archiving job (a background crawler that collects old news) happened to run in that exact window, and misread "network connection failed" as "no news to collect today."
Left alone, it would have marked several days' worth of not-yet-collected data as "fully collected." The root cause was failing to distinguish "the network briefly dropped" from "there's genuinely no data." I added a retry mechanism, plus a safeguard that waits for the network to reconnect before starting right after a reboot.
Also today
- I finally decided to fully retire a reinforcement-learning-based model that had sat for a long time in "not yet verified for live use" status. It never beat the benchmark across several tuning attempts, and re-examining it from a design standpoint this time confirmed the approach was never a good fit for this role to begin with, so I let it go without regret.
- I also found that several core prediction models were trained long enough ago that they were falling behind recent market changes. They've only been retrained manually so far; I decided to add a system for periodic automatic retraining going forward.
Today was about doubting "what I concluded was better yesterday" and digging deeper into it. On the surface it looks like confirming the same conclusion twice, but something new turns up every time I dig in.
Top comments (0)