The most dangerous change I've ever shipped was a cleanup.
No new feature and no risky migration. One line that made the code tidier. On 1 May 2026 it took tunyt's production down thirty seconds after CI went green.
The line was mine, and so is this write-up.
Decision 1: put everything that starts in one place
tunyt does a lot of work in the background: notifications, analytics, AI jobs. That work lives in queues, and next to the queues live sweepers. A sweeper is a small loop that keeps a queue healthy. It finds jobs that got stuck, cleans up old ones and nudges things along.
Our workers all started in one function, startWorker(). The sweepers started somewhere else, and that bothered me. Two places where background work begins means two places to forget something and two places to read when you're debugging at night.
So I moved the sweepers into startWorker(). One line. Lint passed, tests passed, the build passed.
I'd still defend the instinct, because having one place where things start is good engineering. What I didn't do was count.
What that line actually did
startWorker() doesn't just call a function. It gives the work its own worker thread. In Node, a worker thread isn't a lightweight coroutine. It's a separate JavaScript runtime with its own heap and its own event loop, and in our setup its own connections.
So every sweeper that moved in got the full package. On a single node, all at once, that meant:
- 21 V8 runtimes,
- 21 thread pools,
- 21 Redis connections,
booting together on one machine, fighting for the same CPU and memory while the API tried to serve real people. The node didn't slow down gracefully. It just went.
Why CI couldn't have caught it
I think about this part the most.
CI did its job perfectly. It checked that the code was correct. The sweepers still swept, the workers still worked, the types lined up. All true.
What CI never checks is what happens when you start 21 of something on one box at the same moment. That isn't a correctness question. It's a capacity question, and our pipeline had no opinion about capacity. A green pipeline tells you the code does what you wrote. It can't tell you whether you should have written it.
Decision 2: threads only for work that needs them
The fix came from asking what sweepers actually do. They're almost pure waiting. Ask Redis something, wait. Ask the database something, wait.
Node's event loop is built for exactly that. Thousands of things can wait at once on a single thread, each costing almost nothing while it waits. A thread helps when you have CPU work that would otherwise block everything else. For work that mostly waits, a thread buys you nothing and costs you a whole runtime.
So the sweepers went back in-process, on the main event loop, where they belonged. The rule we kept is that only CPU-heavy work gets its own thread.
// Mostly waiting on I/O: stays on the event loop, where waiting is cheap.
startSweepers(queues);
// CPU-heavy: worth a thread, because it would otherwise block everything else.
startWorker("some-cpu-heavy-job");
Decision 3: count before you multiply
Isolation has a price, and you pay it per instance.
A thread, a runtime, a connection pool, a container: each one costs memory and start-up time on its own, and the cost multiplies by however many you create. One extra worker is nothing. Twenty-one at once, on one node, during a deploy, is an outage.
So before I split anything into its own worker, process or service, I write down two numbers: what one instance costs, and how many there will be. If I can't answer both, I'm not ready to make the change.
What changed after
"Does this multiply something?" is now my question for anything that touches how processes, workers or connections start. It's really a version of the question I ask before shipping anything: if 100K users hit this tonight, what breaks first? On 1 May the 100K was my own code.
The cleanup was a good idea. Shipping it without counting was the bug.
Top comments (0)