From a generation that spiraled out of control forever, to a morning fix that quietly halved throughput by evening
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
Spanning the weekend, today was about wrapping up things found over the past few days. They had one thing in common: all three had been going wrong quietly, with no visible sign.
Incident 1: One response ran on forever and stalled production
I noticed one of the overnight analysis jobs oddly stuck. It turned out the local LLM server had no upper limit set on response length. Normally a single response ends quickly, but if the model ever got stuck repeating itself without stopping, it could spiral out to tens of thousands of tokens and tens of minutes. What made it worse was that the monitoring setup couldn't catch this case at all — it only got noticed because someone happened to ask "why is this so slow?"
I capped the response length to stop runaway generation, and while I was at it, designed the progress-monitoring mechanism that was actually needed. One interesting design choice came out of this: rather than tracking "how many minutes has this been running," it tracks "how many minutes since anything was last logged." The original idea was to flag anything running some multiple longer than a ticker's average time, but that risked wrongly killing large tickers that normally take longer, or missing a small ticker that got stuck. Using "silence" itself as the signal works consistently regardless of ticker size, with no per-ticker tuning needed. For now it just alerts in observation mode; once a few days pass with no false positives, I'll turn on the actual kill-switch.
Incident 2: A morning fix quietly halved throughput by evening
A few days ago I widened how much input the model could read at once, to fix a problem where some inputs were too long and getting truncated. That had an unexpected side effect — with the exact same settings, throughput was fast at dawn and nearly 2.7x slower by evening.
The cause: a wider input window uses more GPU memory, and during the day and evening other programs (browser, remote access, other analysis jobs) were already using some GPU memory, leaving little headroom. Once the model no longer fit entirely on the GPU and part of it spilled over to CPU, speed dropped sharply from that point on. At dawn, everything else was idle and the GPU had headroom, so it stayed fast.
I got this wrong twice along the way. I judged one report as "looks duplicated," when it was actually a counting mistake that merged two entries for the same ticker into one. I also brushed off "this ticker is just big, so it's naturally slow" — turns out the same ticker had finished much faster on other days. Relearned the lesson: "whenever you change a setting, you have to actually measure speed before and after." Once again, if nobody had actually measured it, this would have slipped by unnoticed.
Incident 3: A date library missed a newly designated holiday
I found that the external library used to determine market holidays was still treating a day newly designated as a legal holiday this year as a "trading day." This library is widely used and well-tested, but it hadn't caught up in real time with a domestic holiday designation change made earlier this year. Left as-is, that entire day would have gone wrong — the automated analysis running normally, paper-trading verification running on a day with no data, the morning report going out — a chain of errors from multiple places wrongly concluding "yes, that's a trading day." Fortunately I caught it before that day arrived, and worked around it by manually registering the exception date.
This confirmed again that neither an external library nor my own knowledge should be assumed "up to date." Things like domestic rules and schedules that rarely change but occasionally do are especially easy for everyone to miss for a while after they change.
Also today
- The overnight analysis had been missing the overall market mood for the day (e.g. a broad sell-off) by only looking at ticker-specific issues. After discussing it with another AI, I designed it around feeding in only reproducible numbers, rule-based (that day's index return, how much a given ticker moved relative to the market) instead of a text summary — a summary varies slightly each time and can contaminate comparison experiments, while numbers reproduce identically every time.
- Finalized the plan to add another graphics card. A spare card from a different brand was also a candidate, but I ruled it out because a different compute backend risks subtly different results even on the same model, and decided to add another card of the same kind I'm currently using instead — to keep "the hardware itself caused the difference" out of comparison experiments as noise.
The trap in all three was that "nothing looks wrong right now." The runaway only happens once in a while, the slowdown comes and goes with time of day, and the holiday problem shows no sign until the day itself arrives. A day spent catching things that could have quietly slipped by.
Top comments (0)