The code was deployed, but the process meant to run it had already been running since before that
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
The trading system has a scheduling queue for tasks meant to run "not now, but once a specific condition becomes true later." I recently added a new condition to that queue, and the task I registered sat silently for over an hour without firing.
What happened
The new condition was meant to judge "is it currently safe to restart a resident process." This particular restart couldn't overlap with other overnight jobs, so it was designed to wait until the condition became true before triggering the restart.
I deployed the code and registered the task in the queue. But even well past the time window when the condition should have been true, it hadn't fired for over an hour.
This queue re-evaluates its conditions every 5 minutes, so missing it once or twice would be normal. An hour, though, meant the condition had been re-evaluated more than a dozen times.
What I suspected first
My first suspicion was the condition-evaluation logic itself. I wondered whether the function judging "is it safe to restart" was misreading the status of other overnight jobs, incorrectly concluding "not yet" over and over even when it was actually safe.
I went looking for that function's logs — and found no trace of it ever having been called at all. The condition hadn't evaluated to false; it had never been evaluated in the first place.
The real cause
The process running this queue every 5 minutes was a long-lived resident process. And that process had been running before the deployment that added the new condition.
A new condition gets registered in two places in the code: a name string in a whitelist, and a branch that wires that name to the actual judgment logic. The deployment landed on the filesystem fine, but the process that had already loaded that module into memory hours earlier had no idea the change had happened.
From that process's point of view, the new condition's name was simply "unrecognized." The system was designed to lean safe when it saw an unrecognized value — treating it as "condition not met." No error, no exception raised, just a quiet false every single time.
Put together, it was a loop: evaluating the new condition required a process that knew about it, but starting that process fresh was exactly the "safe restart" the condition was meant to protect. The condition was waiting for the restart, and the restart was waiting for the condition to become true — a chicken-and-egg problem.
How I fixed it
The immediate fix was simply restarting that process once, to load the new code into memory. The moment it restarted, the queue evaluated the condition normally and the task fired.
For the actual prevention, I put the fix in process rather than code. I added a hard rule to the deployment checklist: "any deployment that adds a new condition to this queue must either directly confirm it fires right after registration, or restart the resident process that evaluates it at least once to load the new code."
I could have fixed the condition-evaluation logic itself, but the real substance of this incident wasn't a logic bug — it was missing the fact that "deployed" and "the running process knows about the deployment" are two different events. So I chose to close that gap procedurally instead.
The general lesson
When adding a whitelist or dispatch-table entry to a long-lived process, treat "the file changed" and "the process that loaded that file into memory knows about the change" as separate events. This gap hides especially well in interpreted languages. The confidence that "the code is already deployed" is exactly what makes you suspect last that the running process is still on the old version.
"Treat unrecognized values safely as false" is a sound design on its own, but combined with silent failure, it becomes dangerous. Instead of quietly returning false when a value isn't recognized, at minimum log or alert on "this condition name is unknown." That's the only way to tell apart a legitimate false from the judgment logic and a false that came from the system never being able to evaluate the condition at all.
In systems where deployment and process lifecycle are decoupled, it's safer to make "does the process running this also need a restart" an explicit checklist item for every deployment that touches dispatch logic. Rather than relying on someone remembering every time, it would be even better to have the deploy script look at the list of changed files and automatically report back which resident processes import them.
Directly watching a newly registered condition actually fire once, as scheduled, is cheap insurance. In this case, checking whether it fired right after registration would have narrowed down the cause immediately instead of waiting an hour. "It's registered" and "it actually works" are different questions.
Top comments (0)