An overnight stock-scoring batch stalled; fixed the first timeout-less call, then asked whether the fix was enough and found a second one hiding in the same codebase.
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
The overnight batch never ran
During this morning's check, I found that the previous night's stock-scoring batch hadn't run at all.
Tracing it back, one external listing-lookup call had no timeout on it. If the network response never came back, that call would simply hang forever.
The scheduler assumed the job was "already running" and kept skipping the next attempts, which left the scoring results a full day stale.
Fixed it and restarted, fast
The fix was straightforward: wrap the offending call with a timeout, and confirm the fix actually mattered by checking that removing it made the test fail.
I restarted the related resident processes and confirmed the batch started running normally again. After the restart, there were no more skips from the "already running" check.
Asked "is this actually enough?"
Rather than calling it done, I asked another AI tool to review whether this fix was sufficient to prevent a recurrence.
The answer came back as "conditionally, no." There was a second call hitting the same external data source that still had no timeout — the exact same failure mode, just waiting to happen in a different spot.
That meant the same class of bug could still be lurking elsewhere in the codebase. So I wrapped that second call with a timeout the same way and added a test to lock it in.
Added one more safety net
Whether the fix actually holds in production can only be confirmed by watching the next night's batch run cleanly.
Instead of relying on remembering to check that manually, I scheduled a one-shot check for a specific hour the following early morning that inspects the batch's progress and reports back. If everything's fine it stays quiet; if not, it alerts right away.
All within the same day, the loop went: notice the symptom → emergency fix → ask if the fix was sufficient → find a hidden instance of the same bug class → fix that too → add a confirmation safety net. A good reminder that a fast fix is often a starting point, not the finish line.
Top comments (0)