DEV Community: Piyush

Debugging an 11-Second Database Latency in a Cloud Run Background Worker

Piyush — Thu, 30 Jul 2026 13:27:17 +0000

Distributed systems rarely fail in obvious ways.

Recently, I investigated an issue where a NestJS workflow worker running on Google Cloud Run occasionally took 10–11 seconds to resume a workflow after a BullMQ timer fired. The SQL query itself was simple, yet the overall request latency was an order of magnitude higher than expected.

This post walks through the investigation process and, more importantly, the methodology used to isolate the bottleneck.

System Architecture

The worker consisted of:

NestJS
BullMQ + Redis
PostgreSQL (Cloud SQL)
Google Cloud Run

BullMQ Timer
      ↓
Workflow Worker
      ↓
Load Workflow Instance (PostgreSQL)
      ↓
Resume Workflow
      ↓
Persist State

The expectation was that a workflow should resume within a few hundred milliseconds after the timer fired. In practice, latency was bimodal: most resumes were slow but tolerable, and a smaller fraction stalled for 10+ seconds.

Stage	Expected	Typical Observed	Worst-Case Observed
BullMQ receives job	<10 ms	8–15 ms	8–15 ms
Redis lookup	<10 ms	2–6 ms	2–6 ms
PostgreSQL connect + query	<100 ms	1.2–2.7 sec	10–11 sec
Workflow execution	<100 ms	Normal	Normal

The database immediately became the prime suspect, but it turned out not to be the culprit. The "typical" and "worst-case" numbers above turn out to be the same underlying cause at different severities, as Step 4 shows.

Step 1: Instrument Every Layer

Rather than assuming PostgreSQL was slow, I added timing logs around every major component in the execution path:

Timer Fired → BullMQ Worker → Redis → Workflow Service → Repository → Cloud SQL

Each component logged a start timestamp, end timestamp, and elapsed duration.

Component	Average Time
BullMQ job activation	10–30 ms
Redis operations	2–8 ms
Workflow logic	<20 ms
Repository call	1.2 sec – 11 sec

The bottleneck was isolated to the persistence layer, with high variance.

Step 2: Eliminate Application Code

Running the identical worker locally:

Operation	Local
Repository lookup	8–25 ms

No latency, locally, ever. This ruled out the repository implementation, workflow engine, business logic, BullMQ, and Redis. The investigation shifted to infrastructure: Cloud Run configuration, the PostgreSQL connection pool, Cloud SQL, and networking.

Step 3: Verify the Database

Cloud SQL's Query Insights showed consistently fast execution:

Query Time:
42 ms
67 ms
73 ms
58 ms

If Cloud SQL executed the query in under 100 ms, the remaining latency had to occur before the query reached PostgreSQL.

Step 4: Split Connection Time from Query Time

Instead of measuring await repository.findById(id) as one black box, I instrumented the underlying TypeORM QueryRunner directly, and reduced the query itself to SELECT 1 to remove indexes, joins, and execution plans from the picture entirely:

const t0 = Date.now();
await queryRunner.connect();
const acquireMs = Date.now() - t0;

const t1 = Date.now();
await queryRunner.manager.query('SELECT 1');
const queryMs = Date.now() - t1;

logger.log({ acquireMs, queryMs, total: acquireMs + queryMs });

This is what actually redirected the investigation. Two distinct patterns showed up.

Typical sample:

Connection acquisition : 2293 ms
Query execution        :    2 ms
Total                   : 2295 ms

Worst-case pattern, reconstructed from the retry and timeout behavior of the pool, pending an actual 10s+ log sample to swap in:

Attempt 1: connection acquisition (timed out waiting on pool) : 6021 ms
Backoff before retry                                          :  512 ms
Attempt 2: connection acquisition (succeeded)                  : 3844 ms
Query execution                                                :   63 ms
Total                                                          : 10,440 ms

So the 10–11 second cases weren't a separate failure mode. They were the same connection-acquisition stall, occasionally severe enough to blow through the pool's acquire timeout, trigger a retry, and stall a second time before finally succeeding:

Metric	Typical Value	Worst-Case Value
acquireMs (per attempt)	1200–2600 ms	up to 6000 ms
retries	0	1 (+backoff)
queryMs	2–80 ms	2–80 ms

The SQL itself was healthy in every single case. The expensive, variable operation was obtaining a usable PostgreSQL connection, and under worse timing, obtaining it twice.

Step 5: Rule Out PostgreSQL and TypeORM

Another production service was using the same Cloud SQL instance, the same TypeORM configuration, the same credentials, and the same connection pool settings:

Service	acquireMs (typical)	acquireMs (worst-case)
API	0–25 ms	0–40 ms
Worker	1200–2600 ms	up to 6000 ms

If PostgreSQL, Cloud SQL, or the TypeORM configuration were fundamentally at fault, both services would show the same pattern. Only the worker did. The issue was specific to the worker's runtime, not the database layer.

Step 6: Investigate the Runtime

Both services were deployed on Cloud Run. By default, Cloud Run only allocates CPU to an instance while it is actively processing a request. This is "request-based billing," formerly called CPU throttling. Outside of a request, CPU is throttled down, even on an instance that's kept warm.

The API service is request-driven, so it's always in a request when it's doing anything. The worker is different: it spends most of its life idle, waiting on a BullMQ/Redis timer with no inbound HTTP request in flight. min-instances=1 kept the container alive and prevented cold starts, but it did not keep CPU allocated; the two settings are independent. When the timer fired and the worker tried to open a fresh PostgreSQL connection (TCP handshake, TLS negotiation), that work was competing for throttled CPU, exactly the kind of background, non-request-driven operation this billing mode is designed to deprioritize.

I confirmed this directly rather than just inferring it. Cloud Monitoring's container instance count panel shows the instance flip from idle to active at the exact timestamp CPU utilization spikes from ~0% to ~50%. CPU is only granted once the instance becomes active. A same-config SELECT 1 keepalive log from each service, at roughly the same time, makes the gap concrete:

worker-dev:  acquireMs: 2299, queryMs: 100, poolBefore: {waiting: 0}
api:         acquireMs: 26,   queryMs: 2,   poolBefore: {waiting: 0}

waiting: 0 in both rules out pool contention. The worker isn't queued behind other connections; the connect() call itself is just slow under throttled CPU.

The fix was switching the worker's billing/CPU-allocation setting from "only during requests" to "always allocated" (instance-based billing), keeping min-instances=1 so there's always a warm, fully-allocated instance to pick up the timer:

# gcloud
gcloud run services update workflow-worker \
  --region=us-central1 \
  --no-cpu-throttling \
  --min-instances=1 \
  --max-instances=10

# Terraform (google_cloud_run_v2_service)
resource "google_cloud_run_v2_service" "workflow_worker" {
  name     = "workflow-worker"
  location = "us-central1"

  template {
    scaling {
      min_instance_count = 1
      max_instance_count = 10
    }

    containers {
      image = "us-central1-docker.pkg.dev/PROJECT_ID/repo/workflow-worker:latest"

      resources {
        limits = {
          cpu    = "1"
          memory = "512Mi"
        }
        # false = CPU is always allocated (instance-based billing),
        # not just while a request is in flight. This is the
        # Terraform equivalent of `--no-cpu-throttling`.
        cpu_idle = false
      }
    }
  }
}

The trade-off is worth noting: instance-based billing means you pay for the minimum instance continuously, not just while it's handling a request. That's the price of a background worker that reacts to timers instead of HTTP traffic. A request-driven alternative, using Cloud Tasks to invoke the worker over HTTP instead of a persistent BullMQ listener, would let Cloud Run scale to zero between runs, but it's a bigger architectural change and wasn't the right trade-off here.

Results

Metric	Before	After
Connection acquisition (typical)	1200–2600 ms	2–20 ms
Connection acquisition (worst-case)	up to 10,440 ms	2–20 ms
SQL execution	2–80 ms	2–5 ms
Workflow resume latency (p99)	10–11 sec	<200 ms

No application code changed. No SQL optimization was required. No database configuration changed. Only the Cloud Run CPU-allocation setting changed.

Key Engineering Takeaways

Instrument every layer. Without per-component timing, every optimization is based on assumptions.
Separate connection acquisition from query execution. Measuring only repository methods hides where latency actually occurs, and hides that "10 seconds" and "2 seconds" can be the same failure at different severities.
Use the simplest possible workload. Replacing the application query with SELECT 1 eliminated indexes, joins, table size, and execution plans from the investigation.
Compare equivalent systems. Another service using the same Cloud SQL instance became the control group that ruled out PostgreSQL and TypeORM.
Confirm, don't just infer, the runtime cause. A CPU utilization graph over the same time window turned "Cloud Run probably throttled us" into a confirmed diagnosis.
Understand your platform's billing model, not just its scaling model. In serverless environments, whether CPU is allocated during idle time can matter far more than database performance.

Splitting the Monolith: What We Learned Separating Workflows from the Main Server

Piyush — Thu, 16 Jul 2026 17:17:49 +0000

A story about scaling two very different workloads independently — and the one thing that broke when we did.

For a long time, our application ran as a single process. One Node.js server handled everything: HTTP requests, WebSocket connections to browsers, and the workflow engine that lets a process pause and resume later — sending a message, waiting, sending a reminder, and so on.

That worked fine while traffic was small. But two things about that workload didn't sit well together as we grew.

User-facing traffic is bursty and latency-sensitive — someone opens the app, sends a message, expects the interface to respond immediately. Workflow execution is background work with a completely different profile: timers firing minutes or hours later, jobs that need to survive a restart, execution that has nothing to do with any browser currently connected. Bundling both into one process meant we couldn't scale them independently. A spike in workflow volume competed for the same CPU and memory as someone waiting on an API response. Scaling up meant scaling everything, whether it needed it or not.

So we made the call to split them.

The split

We pulled workflow execution into its own process — a dedicated worker, separate from the API, talking to Redis for delayed jobs.

Now the two workloads could scale on their own terms:

The API scales with concurrent users and request volume.
The worker scales with the number of active, in-flight workflows.

Both processes shared the same database, so as long as each did its job correctly, the system as a whole should behave exactly as it did before — just with each half able to grow at its own pace.

That was the theory. Deploying it surfaced something the all-in-one version had been hiding from us for free.

What the split quietly broke

Once workflows ran in their own process, timers still fired exactly when they should. The workflow still resumed. The next chat message still landed in the database, correctly, every time.

But users stopped seeing it. Refreshing the page always showed the message — proving the data was right — yet nothing told the browser to update on its own.

Chasing this, I found the actual cause: the worker had a ChatEventsService, and so did the API — the exact same code, copied over as part of the split. In the old, single-process world, that class used one shared EventEmitter, and any part of the app could react to what any other part had just done.

After the split, that assumption quietly stopped being true. Two processes were now running identical code, importing the same service, calling the same method name — but each with its own EventEmitter, with no memory shared between them at all. The worker was still emitting message.created. It just had nobody left in its own process who needed to hear it, and no way to reach the process that did.

This is the cost that's easy to miss when you split a monolith for scaling reasons: you're not just separating CPU load, you're separating memory. Anything that relied on being in the same process — including things you didn't realize relied on it — stops working silently, not loudly.

Reframing the question

My first instinct was to debug it as a socket problem — was Socket.IO dropping connections, was the frontend failing to re-render. All of that was fine. The real question wasn't about sockets at all:

"How does the knowledge that a message was created travel from the worker to the API, now that they don't share memory?"

Before the split, the answer was "it doesn't have to travel — it's already there." That answer had been invisible because it had never needed to be an answer. Splitting the process for scaling reasons was exactly what turned an implicit assumption into a missing piece of infrastructure.

Giving the two processes a way to talk

Once the split was scaling both workloads the way we wanted, the remaining gap wasn't really about this one worker and this one browser — it was that we now had two independent processes with no explicit channel between them, for anything.

A few options would have patched just this one case:

Option	Why it undoes the split
Move timer execution back into the API	Reintroduces the exact resource contention we split to avoid
Add Socket.IO to the worker	Worker takes on browser-communication it shouldn't own
Poll for updates from the browser	Extra load, worse UX, and doesn't scale either

Each of those either walked the split back or bolted on a one-off fix for this specific path. What we actually needed was a general channel: any process drops a message onto a queue, any other process that cares picks it up — without the two ever needing to share memory again. We introduced a message queue between them, backed by Redis, since we were already depending on Redis for timers.

Ownership stayed exactly where the split put it. The worker still owns workflow execution. The API still owns WebSockets and browsers. We added one explicit path for a fact to cross the boundary we'd just created: the worker drops a message on the queue, the API consumes it, nothing more.

What independent scaling actually cost us

Splitting the monolith gave us what we wanted: the API and the worker now scale on completely different axes, and a burst of workflow activity no longer competes with user-facing request latency. But it wasn't free, and it's worth being honest about the trade:

We gave up implicit communication. Anything that used to "just work" because it lived in shared memory now has to be made explicit, on purpose, for every new interaction between the two processes.
We gained a new failure mode. A message queue is only as reliable as we configure it to be — if we're not careful about acknowledgments and persistence, a message can be dropped if the API is mid-deploy or briefly disconnected at the exact moment something is enqueued.
We took on a new category of problems: making sure messages are delivered reliably, consumed exactly once, processed in the right order, and observable when something goes wrong. None of those existed when everything ran in one process — they're the direct cost of scaling the two halves independently, and they deserve their own write-up rather than a rushed paragraph here.

The takeaway

Splitting a monolith to scale two workloads independently is usually described as an infrastructure decision — more instances of the thing that needs to scale, fewer of the thing that doesn't. That part's true. What's less obvious going in is that the split also quietly removes something you were never billed for: a shared memory space that had been doing communication work on your behalf, for free, the whole time.

I had been assuming that persisting a change and communicating a change were the same operation because, inside a monolith, they often feel that way. Splitting the system forced me to see they're completely different problems. A database answers "What is true?" A messaging system answers "Who needs to know that it became true?" Once I started reasoning with those as separate concerns, the architecture became much easier to understand.

(Refined with AI for readability)

"pod install" Fixed My Expo Package Version (And Taught Me the Difference Between "npm install" and "npx expo install")

Piyush — Thu, 04 Jun 2026 13:21:16 +0000

The Problem That Didn't Make Sense

I was working on an Expo-managed React Native application targeting iOS. Everything looked normal until I noticed a version mismatch involving expo-file-system.

My package.json contained:

{
  "expo": "^54.0.0",
  "expo-file-system": "^55.0.16"
}

However, the Expo SDK 54 documentation showed:

expo-file-system ~19.0.23

At first glance, this looked wrong. How could the package be version 55.0.16 in my project while the documentation recommended 19.0.23?

The confusion became even greater when I ran:

pod install

and saw:

Installing ExpoFileSystem 19.0.23 (was 55.0.16)

The build succeeded.

At that moment I realized I didn't actually understand how Expo packages, native modules, CocoaPods, and version compatibility worked together.

My Initial Assumption

Like many JavaScript developers, I assumed:

npm install expo-file-system

means:

Install the package and everything needed for it.

This mental model works reasonably well for pure JavaScript libraries such as:

npm install axios
npm install zod
npm install date-fns

There is no native code involved. The package version is the package version.

Expo modules are different.

Understanding the Layers

An Expo package actually exists in multiple worlds.

Expo SDK
│
├── JavaScript Package
│   └── expo-file-system
│
├── Native iOS Module
│   └── ExpoFileSystem Pod
│
├── Native Android Module
│   └── Gradle Module
│
└── Expo Compatibility Matrix

When you use:

npm install expo-file-system

npm only knows about the JavaScript package published to npm.

It does not know:

Which Expo SDK you're using
Which React Native version you're using
Which native iOS pod version is compatible
Which Android native implementation should be used

Its job is simply:

Install the latest package matching the version range.

Nothing more.

What `npx expo install` Actually Does

Expo provides its own installation command:

npx expo install expo-file-system

This command behaves differently.

Before installing anything, Expo checks:

Current Expo SDK Version
          ↓
Compatible Package Version
          ↓
Install That Version

In my case:

Expo SDK 54
        ↓
expo-file-system ~19.0.23
        ↓
Install 19.0.23

Instead of grabbing the newest package available on npm, Expo chooses the version tested and validated against the current SDK.

This is why Expo documentation almost always recommends:

npx expo install <expo-package>

instead of:

npm install <expo-package>

Then Why Did `pod install` Fix It?

This was the part that confused me the most.

I expected CocoaPods to simply install whatever was in package.json.

That's not what happens.

During:

pod install

Expo autolinking runs.

Autolinking scans installed Expo modules and reads their native configuration files.

A simplified view looks like:

node_modules
      ↓
Expo Package
      ↓
Podspec
      ↓
CocoaPods
      ↓
Native Pod Installation

When CocoaPods encountered the Expo File System module, it discovered that the native pod version expected for SDK 54 was:

ExpoFileSystem 19.0.23

and corrected the native dependency accordingly.

That's why I saw:

Installing ExpoFileSystem 19.0.23 (was 55.0.16)

The native side was effectively protecting itself from an incompatible version.

The Important Lesson

The successful build was actually misleading.

The build succeeding does not mean the dependency setup is healthy.

I still had a mismatch between:

JavaScript Layer
expo-file-system 55.x

and

Native Layer
ExpoFileSystem 19.x

A future build, CI pipeline, runtime feature, or TypeScript API could easily break because the JavaScript package and native implementation are expecting different things.

The fact that CocoaPods repaired the native side doesn't mean the project is correctly configured.

The Rule I Follow Now

Whenever I add an Expo package:

npx expo install expo-file-system
npx expo install expo-camera
npx expo install expo-location

Whenever I add a normal JavaScript library:

npm install axios
npm install zod
npm install lodash

A simple way to remember it is:

If the package belongs to the Expo ecosystem, let Expo decide the version. If it's a pure JavaScript dependency, let npm decide.

A Helpful Command I Learned

If you inherit an existing Expo project and suspect version drift, run:

npx expo install --fix

Expo will inspect installed Expo packages and align them with the currently installed SDK.

Follow it with:

npx expo doctor

to identify remaining compatibility issues.

Final Mental Model

The mental model that finally made everything click for me was:

npm install
    =
Install package

expo install
    =
Find compatible version
+
Install package

pod install
    =
Install native iOS dependencies

Each tool solves a different problem.

The mistake is assuming npm understands Expo SDK compatibility. It doesn't.

Expo does.

And that small distinction is the reason a seemingly mysterious pod install ended up teaching me how Expo's dependency system actually works.