Tracked down why a setting that won on benchmark lost in production, and closed out a statistical alert I'd been re-litigating for days
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
Trying to run a local LLM through 100 stocks in 10 hours
I spent most of today running an open-source large language model locally, on the stock-analysis pipeline, using my main GPU together with a spare card I picked up recently.
This model uses a mixture-of-experts structure — a large total parameter count, but a much smaller set of parameters actually active during inference. The goal was to process 100 stocks in under 10 hours.
Running requests one at a time already came in over 40% past that target. I expected batching multiple requests together to help, but instead it either slowed things down or made the GPU stop responding entirely.
At first I wrote those failures off as transient network errors. Asking the AI to dig through the system logs showed otherwise: the GPU itself was actually hanging, a hardware-level fault. From then on, I kept concurrency experiments within a range where that fault didn't reproduce.
A separate benchmark confirmed the batching engine itself wasn't at fault. The real bottlenecks were two structural things: the initial prompt-processing stage hogging its turn and stalling other requests behind it, and memory-reference range growing continuously as conversations got longer.
The most promising fix looked like moving part of the computation (the expert layers) entirely onto the secondary GPU. In single-request benchmarks this was clearly faster, and the gain grew the longer the request got.
But re-testing it under real concurrent load flipped the result completely. The setting that won on benchmark was actually slower in production.
I asked the AI to diagnose it again. Raw decoding speed was still faster with this setup, but the initial prompt-processing stage lost out in exactly the opposite way — switching back and forth between the two GPUs happened far more often, and an optimization that overlaps multiple requests' pipelines was disabled entirely under this configuration.
So I dropped that approach and went back to the simpler one — splitting computation by layer, spread evenly across both cards — which turned out to be the best option after all.
Instead, I found gains on a different axis. I applied a patch that reorders prompts so multiple stocks can share a common prefix, and another that nudges the model away from unnecessarily verbose answers, both of which cut processing time.
I couldn't yet tell whether these patches affect the actual stock ratings themselves or just fall within normal run-to-run variance, so I held off on a final verdict there.
I also reviewed the model's compression (quantization) level separately. The level I'm currently using turned out to be fairly aggressive. Going up a level for better quality would exceed GPU memory and slow things down significantly, so I decided against that — instead I'll pull a middle-ground level and run a small side-by-side comparison.
By the end of the day: I'd reached the best result server tuning alone could give, getting close to the 10-hour target but still short of it. Closing the remaining gap needs further code changes, which I'll pursue separately after getting approval.
Closing out a statistical alert after days of re-litigation
I finally closed out a statistical anomaly alert today that I'd been re-examining for days.
Switching to a different model recently increased the number of nightly comparison samples. But the alert's threshold was still calibrated on the old absolute-count basis.
That meant the alert could trip easily even when nothing was actually wrong. Recomputing it as a ratio instead showed the recent activity wasn't statistically distinguishable from the normal baseline.
I closed this case with no action taken, and I'm redefining the alert threshold to use a ratio going forward instead of an absolute count.
While investigating, I also found and fixed two bugs in an observation-only tracking path (used for measuring predictive signal quality). Neither touched live trading logic, so they had no bearing on this verdict.
Also wrapped up today
A few days ago, a motherboard swap and reboot briefly set the system clock to the past; the moment the time server corrected it, the system mistook a schedule that hadn't actually passed yet for one that had, and triggered a scheduled retraining job on a day it wasn't supposed to run.
Today I added two safeguards to the retraining pipeline as follow-up: one that automatically demotes an out-of-schedule run to a dry run instead of a real deployment, and one that holds off deployment if it comes too soon after the last one.
I also retuned the power limit on the spare GPU downward to deal with fan noise, and finished a head-to-head comparison of two driver backends on that card — the existing one won, so that's confirmed as the pick going forward.
With the recent full hardware swap now reflected in the docs, a few old hardware verdicts had their underlying assumptions disappear entirely, so I marked them invalid.
Top comments (0)