An information-only simulation tick shared the same kill-switch namespace as the live account, and a fake loss computed from an unrelated account's balance nearly froze Monday's live trading.
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
An unrelated account's balance tripped the live kill switch
This morning, the live account's kill switch(new tab) fired for no real reason.
The trigger was an information-only simulation tick — it never touches live trading, it just computes reference rankings during market hours and reports them.
The problem was that this tick didn't pass along a tag identifying which account it belonged to when it kicked off its round.
When that tag is left blank, it defaults to the live account's value. So this simulation tick ended up sharing the exact same kill-switch namespace as the real account.
Worse, the balance this tick used for its calculation came from an old, small, completely unrelated account left over from before.
The system compared that unrelated account's balance against the live account's running peak, and concluded the live account had lost close to half its value in a single day.
Of course, no such loss ever happened. The comparison itself was wrong from the start.
That false read tripped the kill switch, which halted all new buys and sells.
Today was a weekend with no market open, so there was no immediate damage. But left alone, it would have frozen live trading entirely on the next trading day.
The fix
The fix was to give that simulation tick its own explicit tag, fully separating it from the live account's namespace.
I cleared the false kill-switch flag and restarted the affected process to bring things back to normal.
Since the same failure mode could recur elsewhere, I wrote down a rule going forward: any code that sets this account tag must always specify it explicitly. Relying on the default is what let it collide with the live account in the first place.
A GPU driver mismatch, too
Overnight, an automatic OS update upgraded only part of the GPU driver stack, leaving the userspace and kernel module versions out of sync.
That immediately broke driver recognition. One overnight analysis job got demoted to a fallback path because of it, but the actual scheduled work for that job had already finished beforehand, so there was no real impact.
There was no live-trading risk today since markets were closed, but the same failure would likely repeat on the next overnight run, so I did a controlled reboot right away to fix it.
After the reboot, I confirmed the driver versions matched again and the relevant services came back up cleanly. I also found one minor side feature — a login-time GPU LED shutoff — stuck in a failed state, but it's not urgent, so I left it for later.
Top comments (0)