DEV Community

finaltype
finaltype

Posted on Originally published at finaltype.github.io

Token Ping-Pong and an Overnight Batch Hang - Two Root-Cause Hunts

Tracked down two operational incidents this week - an auth token that kept invalidating itself, and a nightly batch job that silently hung overnight - and set up automated checks to confirm the fixes actually held.

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

A quiet week, two incidents

This week had fewer devlog entries than usual. Only two went up, on November 2nd and 4th.

Neither was planned feature work — both were incident responses triggered by something breaking in production. And both followed the same pattern: don't stop once the symptom goes away.

Token ping-pong - not restart timing, but a missing cache check

Early in the week, logs showed an external quote API's auth token getting invalidated repeatedly, mostly in the early morning hours.

At first I treated it as a one-off caused by two resident processes restarting at slightly different times and just restarted them. A few days later the same symptom came back, so it needed a real answer this time. I asked another AI tool to read through the actual code, and the real cause turned out to be different: whenever the API rejected a token, the client immediately requested a brand new one, without first checking whether another process had already fetched a fresh one.

Several processes using this API run at the same time. When one gets rejected and issues a new token, that instantly invalidates whatever the other process was holding — and the other process reacts the same way, invalidating the first one right back. That's the ping-pong: two processes repeatedly invalidating each other's tokens.

The fix was straightforward: on rejection, re-read the stored cache first and retry with it if it had already changed, and only issue a brand new token — under a file lock, one process at a time — if that still failed. It was implemented and tested in an isolated workspace, then merged, and all six resident processes using this API were restarted to confirm a clean restart.

Rather than just declaring it fixed, I built a small script that tallies invalidation events from the logs — dozens of events before the fix, zero after deployment. I also scheduled this same tally to run automatically on upcoming trading days and report over Telegram, instead of relying on remembering to check back manually.

A separate statistical hypothesis test also reached its conclusion the same day. A candidate signal based on supply-demand flow data was tested against a procedure fixed in advance, and it failed to clear the statistical bar in its first observation window. Rather than lowering the bar or reshuffling the window to try again, I closed the hypothesis exactly per the pre-registered rule.

Overnight batch hang - the turning point was asking "is this fix actually enough?"

On November 4th, a morning check revealed that the previous night's stock-evaluation batch hadn't run at all.

The cause was an external listing-lookup call with no timeout. If the network response never came back, that call would hang forever, and the scheduler would just assume the job was "already running" and keep skipping it.

I wrapped the problematic call in a timeout, and verified the fix actually mattered by confirming tests failed when the fix was removed. All related resident processes were restarted and confirmed to run normally again.

The turning point was not stopping there — I asked another AI to review whether the fix was sufficient to prevent recurrence. The answer came back: "not unconditionally."

There was a second call using the same external data source that still had no timeout — the exact same failure mode, just not yet triggered. That meant the same class of defect could be hiding elsewhere in the codebase, so that call got the same timeout treatment and a new test to lock it in.

Whether the fix actually holds in production can only be confirmed by watching the next night's batch run cleanly. Instead of relying on remembering to check, I scheduled a one-time automated check for a specific early-morning hour the next day, to report the batch's status.

Why it matters

What ran through both incidents this week was the same attitude: a fix is a starting point, not an ending point.

For the token ping-pong, I didn't stop once a restart made the symptom go away — I waited for it to recur and had another AI read the code to find the real root cause: the missing cache check.

For the overnight batch hang, I didn't stop once the timeout made the symptom disappear either — asking "is this fix sufficient?" again surfaced a second hidden instance of the same bug class.

In both cases, instead of trusting memory to catch a recurrence, an automated check got deployed right alongside the fix, so that if the problem comes back, it gets noticed immediately.

Top comments (0)