Series — Building ACH on AWS: 0: System Design · 1: Sidekiq Latency · 2: Extraction Boundary · 3: Shadow Traffic · 4: What Broke · 5: Lambda/ECS Split · 6: SQS FIFO · 7: Instrumentation · 8: Triage Panels
The alert came in around 8am on a weekday. Web request latency had climbed
sharply, not spiked and recovered, but climbed and stayed. No recent
deploys. ProfitStars was healthy. Puma logs showed nothing unusual. The
error rate was flat.
It took us an embarrassingly long time to notice the pattern: the latency
degradation was happening every morning around the same time. And every
morning around the same time, the T+1 ACH settlement batch arrived.
That is the story of how we discovered that our Sidekiq workers and our
Puma web workers were quietly fighting over the same resources, and why
fixing it required pulling ACH out of the monolith entirely.
What ACH looked like in the monolith
If you read the previous post in this series, you already know how the
settlement state machine and allocator worked, so I will not retrace
that here. The short version: a law firm's client pays an ACH invoice,
a payment record enters "submitted" state, and confirmation arrives via
async callback from ProfitStars one to two business days later.
In the monolith, that whole flow lived in two places. Outbound ACH
submissions were Sidekiq background jobs, enqueued by the checkout
controller after a payment record was created. Inbound ProfitStars
settlement callbacks posted to a standard Rails webhook controller, which
ran state machine transitions and the allocator inline before returning
200 to the processor.
Both the Sidekiq workers and the Puma web processes lived on the same
application servers. They shared a database connection pool that had been
sized for web request concurrency (the expected simultaneous Puma thread
count), not for the burst profile of a settlement batch.
The queue configuration made things worse. ACH jobs ran in a shared queue
alongside email delivery, billing calculations, and report generation:
no priority isolation, no dedicated ACH concurrency cap. When
ACH work arrived, it competed for the same worker slots as everything else.
Why this worked fine until it did not
For most of the day, the shared setup was invisible as a problem. ACH
submissions trickled in throughout business hours. Each one enqueued a
job, the job ran, ProfitStars returned an acknowledgment, and the worker
finished. Settlement callbacks arrived on roughly the same cadence,
spread across the day in proportion to when submissions had gone out.
ACH settlement is a batch process, though. ProfitStars does not confirm
individual payments as they settle. It processes the previous day's
submissions overnight and delivers a settlement batch. That batch posts
back to our webhook endpoint the next morning (T+1), and it arrives
all at once.
Every firm that had accepted an ACH payment the previous business day had
its settlement callback arrive in a tight window. Our Sidekiq workers
spiked to handle the burst. On a shared-host setup, that spike directly
reduced the process capacity available to Puma. Web request latency
climbed because Puma threads were competing with ACH settlement workers
for CPU, memory, and database connections.
The database connection pool was the sharpest constraint. The pool had
been sized to service the expected number of concurrent web requests. At
8am, settlement callbacks were driving Sidekiq worker concurrency to its
peak at the same moment web traffic was also near its morning peak. Both
pools pulled from the same PostgreSQL connection limit. Workers queued
waiting for connections. That queue depth added latency to every request
that needed the database.
We could see this in hindsight: process-level CPU contention in our host
metrics, connection wait times rising in query logs, and the strict
daily regularity of the pattern pointing directly at the settlement batch
window. When we correlated the degradation timestamps with ProfitStars
batch arrival times, the match was exact.
What we missed by not isolating earlier
The operational consequence was predictable once we saw it: a system built
for two different load profiles (web request traffic and async payment
settlement bursts) was being run as if they had the same resource shape.
They do not.
Web traffic scales with user activity, peaks during business hours, and
distributes relatively evenly across the day. ACH settlement is a batch
that arrives in a compressed window at a predictable time, hits a burst
load, and then goes quiet until the next batch. An always-on Sidekiq
worker pool sized to handle the settlement burst was carrying capacity it
did not need for the other 23 hours of the day.
Running both workloads on the same hosts meant neither could be sized
independently. The instances had to be large enough to handle the
combination of web peak and ACH settlement peak simultaneously. That
combination never happened intentionally; it happened because there was
no isolation preventing it.
The right-sizing opportunity was significant. Once we extracted the ACH
workers to their own isolated fleet, we could scale the web tier to its
actual traffic profile, not the worst-case combined profile. That
difference was the source of roughly 30% of the infrastructure cost
reduction we saw after the extraction completed.
Where the extraction pointed
The contention problem had a straightforward diagnosis: co-location of
workloads with incompatible burst profiles, no isolation between them, and
a shared resource constraint that made the collision show up in latency
rather than errors. That made it easy to defer until it was impossible to
ignore.
The fix was not "add more Sidekiq concurrency" or "increase the connection
pool." Both of those would have delayed the problem without removing it.
The right answer was to stop treating ACH settlement as just another
background job that happened to be important, and to give it a runtime
that matched its actual operational requirements: isolated, scalable
independently, and capable of handling batch bursts without affecting the
web tier.
That meant extraction to its own ECS-hosted worker fleet. The settlement
callbacks that previously hit the monolith webhook controller would route
through Lambda (stateless, immediately scalable) to an SQS queue that
the ECS workers consumed. The web tier would never see the settlement
burst.
That is the architecture we landed on, and the rest of this series covers
how we got there. But the harder question (the one we had to answer
before writing a line of extraction code) was where exactly to cut. Not
every ACH concern belonged in the extracted service. The state machine,
the allocator, and the ledger writes had reasons to stay in the monolith,
and getting that boundary wrong would have created a different class of
problem entirely.
Part 2 covers how we found that boundary.
Top comments (0)