DEV Community: Piyush

Splitting the Monolith: What We Learned Separating Workflows from the Main Server

Piyush — Thu, 16 Jul 2026 17:17:49 +0000

A story about scaling two very different workloads independently — and the one thing that broke when we did.

For a long time, our application ran as a single process. One Node.js server handled everything: HTTP requests, WebSocket connections to browsers, and the workflow engine that lets a process pause and resume later — sending a message, waiting, sending a reminder, and so on.

That worked fine while traffic was small. But two things about that workload didn't sit well together as we grew.

User-facing traffic is bursty and latency-sensitive — someone opens the app, sends a message, expects the interface to respond immediately. Workflow execution is background work with a completely different profile: timers firing minutes or hours later, jobs that need to survive a restart, execution that has nothing to do with any browser currently connected. Bundling both into one process meant we couldn't scale them independently. A spike in workflow volume competed for the same CPU and memory as someone waiting on an API response. Scaling up meant scaling everything, whether it needed it or not.

So we made the call to split them.

The split

We pulled workflow execution into its own process — a dedicated worker, separate from the API, talking to Redis for delayed jobs.

Now the two workloads could scale on their own terms:

The API scales with concurrent users and request volume.
The worker scales with the number of active, in-flight workflows.

Both processes shared the same database, so as long as each did its job correctly, the system as a whole should behave exactly as it did before — just with each half able to grow at its own pace.

That was the theory. Deploying it surfaced something the all-in-one version had been hiding from us for free.

What the split quietly broke

Once workflows ran in their own process, timers still fired exactly when they should. The workflow still resumed. The next chat message still landed in the database, correctly, every time.

But users stopped seeing it. Refreshing the page always showed the message — proving the data was right — yet nothing told the browser to update on its own.

Chasing this, I found the actual cause: the worker had a ChatEventsService, and so did the API — the exact same code, copied over as part of the split. In the old, single-process world, that class used one shared EventEmitter, and any part of the app could react to what any other part had just done.

After the split, that assumption quietly stopped being true. Two processes were now running identical code, importing the same service, calling the same method name — but each with its own EventEmitter, with no memory shared between them at all. The worker was still emitting message.created. It just had nobody left in its own process who needed to hear it, and no way to reach the process that did.

This is the cost that's easy to miss when you split a monolith for scaling reasons: you're not just separating CPU load, you're separating memory. Anything that relied on being in the same process — including things you didn't realize relied on it — stops working silently, not loudly.

Reframing the question

My first instinct was to debug it as a socket problem — was Socket.IO dropping connections, was the frontend failing to re-render. All of that was fine. The real question wasn't about sockets at all:

"How does the knowledge that a message was created travel from the worker to the API, now that they don't share memory?"

Before the split, the answer was "it doesn't have to travel — it's already there." That answer had been invisible because it had never needed to be an answer. Splitting the process for scaling reasons was exactly what turned an implicit assumption into a missing piece of infrastructure.

Giving the two processes a way to talk

Once the split was scaling both workloads the way we wanted, the remaining gap wasn't really about this one worker and this one browser — it was that we now had two independent processes with no explicit channel between them, for anything.

A few options would have patched just this one case:

Option	Why it undoes the split
Move timer execution back into the API	Reintroduces the exact resource contention we split to avoid
Add Socket.IO to the worker	Worker takes on browser-communication it shouldn't own
Poll for updates from the browser	Extra load, worse UX, and doesn't scale either

Each of those either walked the split back or bolted on a one-off fix for this specific path. What we actually needed was a general channel: any process drops a message onto a queue, any other process that cares picks it up — without the two ever needing to share memory again. We introduced a message queue between them, backed by Redis, since we were already depending on Redis for timers.

Ownership stayed exactly where the split put it. The worker still owns workflow execution. The API still owns WebSockets and browsers. We added one explicit path for a fact to cross the boundary we'd just created: the worker drops a message on the queue, the API consumes it, nothing more.

What independent scaling actually cost us

Splitting the monolith gave us what we wanted: the API and the worker now scale on completely different axes, and a burst of workflow activity no longer competes with user-facing request latency. But it wasn't free, and it's worth being honest about the trade:

We gave up implicit communication. Anything that used to "just work" because it lived in shared memory now has to be made explicit, on purpose, for every new interaction between the two processes.
We gained a new failure mode. A message queue is only as reliable as we configure it to be — if we're not careful about acknowledgments and persistence, a message can be dropped if the API is mid-deploy or briefly disconnected at the exact moment something is enqueued.
We took on a new category of problems: making sure messages are delivered reliably, consumed exactly once, processed in the right order, and observable when something goes wrong. None of those existed when everything ran in one process — they're the direct cost of scaling the two halves independently, and they deserve their own write-up rather than a rushed paragraph here.

The takeaway

Splitting a monolith to scale two workloads independently is usually described as an infrastructure decision — more instances of the thing that needs to scale, fewer of the thing that doesn't. That part's true. What's less obvious going in is that the split also quietly removes something you were never billed for: a shared memory space that had been doing communication work on your behalf, for free, the whole time.

I had been assuming that persisting a change and communicating a change were the same operation because, inside a monolith, they often feel that way. Splitting the system forced me to see they're completely different problems. A database answers "What is true?" A messaging system answers "Who needs to know that it became true?" Once I started reasoning with those as separate concerns, the architecture became much easier to understand.

(Refined with AI for readability)

"pod install" Fixed My Expo Package Version (And Taught Me the Difference Between "npm install" and "npx expo install")

Piyush — Thu, 04 Jun 2026 13:21:16 +0000

The Problem That Didn't Make Sense

I was working on an Expo-managed React Native application targeting iOS. Everything looked normal until I noticed a version mismatch involving expo-file-system.

My package.json contained:

{
  "expo": "^54.0.0",
  "expo-file-system": "^55.0.16"
}

However, the Expo SDK 54 documentation showed:

expo-file-system ~19.0.23

At first glance, this looked wrong. How could the package be version 55.0.16 in my project while the documentation recommended 19.0.23?

The confusion became even greater when I ran:

pod install

and saw:

Installing ExpoFileSystem 19.0.23 (was 55.0.16)

The build succeeded.

At that moment I realized I didn't actually understand how Expo packages, native modules, CocoaPods, and version compatibility worked together.

My Initial Assumption

Like many JavaScript developers, I assumed:

npm install expo-file-system

means:

Install the package and everything needed for it.

This mental model works reasonably well for pure JavaScript libraries such as:

npm install axios
npm install zod
npm install date-fns

There is no native code involved. The package version is the package version.

Expo modules are different.

Understanding the Layers

An Expo package actually exists in multiple worlds.

Expo SDK
│
├── JavaScript Package
│   └── expo-file-system
│
├── Native iOS Module
│   └── ExpoFileSystem Pod
│
├── Native Android Module
│   └── Gradle Module
│
└── Expo Compatibility Matrix

When you use:

npm install expo-file-system

npm only knows about the JavaScript package published to npm.

It does not know:

Which Expo SDK you're using
Which React Native version you're using
Which native iOS pod version is compatible
Which Android native implementation should be used

Its job is simply:

Install the latest package matching the version range.

Nothing more.

What `npx expo install` Actually Does

Expo provides its own installation command:

npx expo install expo-file-system

This command behaves differently.

Before installing anything, Expo checks:

Current Expo SDK Version
          ↓
Compatible Package Version
          ↓
Install That Version

In my case:

Expo SDK 54
        ↓
expo-file-system ~19.0.23
        ↓
Install 19.0.23

Instead of grabbing the newest package available on npm, Expo chooses the version tested and validated against the current SDK.

This is why Expo documentation almost always recommends:

npx expo install <expo-package>

instead of:

npm install <expo-package>

Then Why Did `pod install` Fix It?

This was the part that confused me the most.

I expected CocoaPods to simply install whatever was in package.json.

That's not what happens.

During:

pod install

Expo autolinking runs.

Autolinking scans installed Expo modules and reads their native configuration files.

A simplified view looks like:

node_modules
      ↓
Expo Package
      ↓
Podspec
      ↓
CocoaPods
      ↓
Native Pod Installation

When CocoaPods encountered the Expo File System module, it discovered that the native pod version expected for SDK 54 was:

ExpoFileSystem 19.0.23

and corrected the native dependency accordingly.

That's why I saw:

Installing ExpoFileSystem 19.0.23 (was 55.0.16)

The native side was effectively protecting itself from an incompatible version.

The Important Lesson

The successful build was actually misleading.

The build succeeding does not mean the dependency setup is healthy.

I still had a mismatch between:

JavaScript Layer
expo-file-system 55.x

and

Native Layer
ExpoFileSystem 19.x

A future build, CI pipeline, runtime feature, or TypeScript API could easily break because the JavaScript package and native implementation are expecting different things.

The fact that CocoaPods repaired the native side doesn't mean the project is correctly configured.

The Rule I Follow Now

Whenever I add an Expo package:

npx expo install expo-file-system
npx expo install expo-camera
npx expo install expo-location

Whenever I add a normal JavaScript library:

npm install axios
npm install zod
npm install lodash

A simple way to remember it is:

If the package belongs to the Expo ecosystem, let Expo decide the version. If it's a pure JavaScript dependency, let npm decide.

A Helpful Command I Learned

If you inherit an existing Expo project and suspect version drift, run:

npx expo install --fix

Expo will inspect installed Expo packages and align them with the currently installed SDK.

Follow it with:

npx expo doctor

to identify remaining compatibility issues.

Final Mental Model

The mental model that finally made everything click for me was:

npm install
    =
Install package

expo install
    =
Find compatible version
+
Install package

pod install
    =
Install native iOS dependencies

Each tool solves a different problem.

The mistake is assuming npm understands Expo SDK compatibility. It doesn't.

Expo does.

And that small distinction is the reason a seemingly mysterious pod install ended up teaching me how Expo's dependency system actually works.