Two failure types that grade distributions alone couldn't reveal, plus a structured-output problem fixed by migrating frameworks
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
It's still early days for this project, so a lot changes day to day. I probably won't post every day, but on days with a substantial change I'll leave a short note like this one. Today's work fell into three areas.
1. A formal comparison of three local LLMs
I needed to pick a local model for a multi-agent LLM research pipeline, so I ran three candidates through the same conditions — same ticker list, same dates — one after another.
The comparison had two stages.
- Automated checks: failure rate, whether final grades skewed to one side, whether unrelated ticker names leaked in, and time spent.
- Manual spot checks: opening the actual generated reports one by one to see if the completeness held up and whether the grade actually matched the reasoning.
This is where it got interesting. Two problems showed up that surface-level metrics alone couldn't catch.
- One model had most of its final grades pile up on "Hold" — but not because it was cautious. The step that generates the final judgment sentence was failing outright at a fairly high rate, and the empty result was defaulting to that grade. Looking only at the grade-distribution table would have led to "this model is just conservative" and nothing more.
- Another model looked clean on every metric, but cross-checking the ticker names revealed a low-probability hallucination where it analyzed a completely different company and drew a conclusion from that. I judged this far quieter and far more dangerous than a skewed grade distribution, and dropped the model.
Both candidates ended up rejected, and I kept the existing model. It was a good reminder that metrics alone aren't enough — you have to open up the actual output.
2. Migrating the LLM framework itself to upstream
I moved the framework used for multi-agent research off a custom fork and onto the original open-source project. The trigger was a problem I hit while testing with a lighter model: structured output (JSON, etc.) kept failing to parse at the final-judgment step.
Tracing the cause, I found a known issue where the local LLM server's (Ollama) OpenAI-compatible API effectively ignores "please answer only in this format" requests. The framework was calling the model through that path, so whenever the model answered in free text instead of the required format, it failed outright.
The workaround was to use the same server's native API instead, which can enforce the output format at the grammar level. Running the same test, structured output had failed every time before the migration and only barely passed by falling back to free text — after the migration, it passed reliably.
3. Polishing the auto-invest feature, and catching a bug before deploy
I refined the feature that automatically splits an amount of money across AI-recommended tickers. I made the allocation method a bit more precise, and added a safety check that automatically holds off on a buy — and flags it separately — if the real-time bid-ask spread looks abnormally wide right before the purchase (a sign of a liquidity problem).
I also touched up some of the rebalancing-alert logic, and in the process caught a potential bug before deploying it. While changing the alert conditions, I nearly missed that the actual order-placement code was still referencing the old calculation method. I caught it during review and fixed it before it ever hit the live account.
4. Two infrastructure bugs
- The logic that checks whether a heavy computation job is using the GPU had a gap: in one specific situation (a temporary benchmark run), it missed the check entirely and caused an actual crash from GPU memory exhaustion. There was already a general-purpose check function elsewhere in the codebase; this particular script just wasn't using it. Applying it there closed the gap.
- One local LLM kept holding onto GPU memory even after finishing a report. It was already making an explicit cache-clearing call, but the memory still didn't release. The root cause is still unclear, but I confirmed that restarting the process reliably reclaims it, and added a safeguard that checks available memory before starting new work and restarts the process if needed.
Also today
Tightened the file permissions on .env and other sensitive config files.
That's it for today. I'll keep leaving these short notes whenever something notable comes up.
Top comments (0)