A story about scaling two very different workloads independently — and the one thing that broke when we did.
For a long time, our application ran as a single process. One Node.js server handled everything: HTTP requests, WebSocket connections to browsers, and the workflow engine that lets a process pause and resume later — sending a message, waiting, sending a reminder, and so on.
That worked fine while traffic was small. But two things about that workload didn't sit well together as we grew.
User-facing traffic is bursty and latency-sensitive — someone opens the app, sends a message, expects the interface to respond immediately. Workflow execution is background work with a completely different profile: timers firing minutes or hours later, jobs that need to survive a restart, execution that has nothing to do with any browser currently connected. Bundling both into one process meant we couldn't scale them independently. A spike in workflow volume competed for the same CPU and memory as someone waiting on an API response. Scaling up meant scaling everything, whether it needed it or not.
So we made the call to split them.
The split
We pulled workflow execution into its own process — a dedicated worker, separate from the API, talking to Redis for delayed jobs.
Now the two workloads could scale on their own terms:
- The API scales with concurrent users and request volume.
- The worker scales with the number of active, in-flight workflows.
Both processes shared the same database, so as long as each did its job correctly, the system as a whole should behave exactly as it did before — just with each half able to grow at its own pace.
That was the theory. Deploying it surfaced something the all-in-one version had been hiding from us for free.
What the split quietly broke
Once workflows ran in their own process, timers still fired exactly when they should. The workflow still resumed. The next chat message still landed in the database, correctly, every time.
But users stopped seeing it. Refreshing the page always showed the message — proving the data was right — yet nothing told the browser to update on its own.
Chasing this, I found the actual cause: the worker had a ChatEventsService, and so did the API — the exact same code, copied over as part of the split. In the old, single-process world, that class used one shared EventEmitter, and any part of the app could react to what any other part had just done.
After the split, that assumption quietly stopped being true. Two processes were now running identical code, importing the same service, calling the same method name — but each with its own EventEmitter, with no memory shared between them at all. The worker was still emitting message.created. It just had nobody left in its own process who needed to hear it, and no way to reach the process that did.
This is the cost that's easy to miss when you split a monolith for scaling reasons: you're not just separating CPU load, you're separating memory. Anything that relied on being in the same process — including things you didn't realize relied on it — stops working silently, not loudly.
Reframing the question
My first instinct was to debug it as a socket problem — was Socket.IO dropping connections, was the frontend failing to re-render. All of that was fine. The real question wasn't about sockets at all:
"How does the knowledge that a message was created travel from the worker to the API, now that they don't share memory?"
Before the split, the answer was "it doesn't have to travel — it's already there." That answer had been invisible because it had never needed to be an answer. Splitting the process for scaling reasons was exactly what turned an implicit assumption into a missing piece of infrastructure.
Giving the two processes a way to talk
Once the split was scaling both workloads the way we wanted, the remaining gap wasn't really about this one worker and this one browser — it was that we now had two independent processes with no explicit channel between them, for anything.
A few options would have patched just this one case:
| Option | Why it undoes the split |
|---|---|
| Move timer execution back into the API | Reintroduces the exact resource contention we split to avoid |
| Add Socket.IO to the worker | Worker takes on browser-communication it shouldn't own |
| Poll for updates from the browser | Extra load, worse UX, and doesn't scale either |
Each of those either walked the split back or bolted on a one-off fix for this specific path. What we actually needed was a general channel: any process drops a message onto a queue, any other process that cares picks it up — without the two ever needing to share memory again. We introduced a message queue between them, backed by Redis, since we were already depending on Redis for timers.
Ownership stayed exactly where the split put it. The worker still owns workflow execution. The API still owns WebSockets and browsers. We added one explicit path for a fact to cross the boundary we'd just created: the worker drops a message on the queue, the API consumes it, nothing more.
What independent scaling actually cost us
Splitting the monolith gave us what we wanted: the API and the worker now scale on completely different axes, and a burst of workflow activity no longer competes with user-facing request latency. But it wasn't free, and it's worth being honest about the trade:
- We gave up implicit communication. Anything that used to "just work" because it lived in shared memory now has to be made explicit, on purpose, for every new interaction between the two processes.
- We gained a new failure mode. A message queue is only as reliable as we configure it to be — if we're not careful about acknowledgments and persistence, a message can be dropped if the API is mid-deploy or briefly disconnected at the exact moment something is enqueued.
- We took on a new category of problems: making sure messages are delivered reliably, consumed exactly once, processed in the right order, and observable when something goes wrong. None of those existed when everything ran in one process — they're the direct cost of scaling the two halves independently, and they deserve their own write-up rather than a rushed paragraph here.
The takeaway
Splitting a monolith to scale two workloads independently is usually described as an infrastructure decision — more instances of the thing that needs to scale, fewer of the thing that doesn't. That part's true. What's less obvious going in is that the split also quietly removes something you were never billed for: a shared memory space that had been doing communication work on your behalf, for free, the whole time.
I had been assuming that persisting a change and communicating a change were the same operation because, inside a monolith, they often feel that way. Splitting the system forced me to see they're completely different problems. A database answers "What is true?" A messaging system answers "Who needs to know that it became true?" Once I started reasoning with those as separate concerns, the architecture became much easier to understand.
(Refined with AI for readability)
Top comments (0)