Series — Building ACH on AWS: 0: System Design · 1: Sidekiq Latency · 2: Extraction Boundary · 3: Shadow Traffic · 4: What Broke · 5: Lambda/ECS Split · 6: SQS FIFO · 7: Instrumentation · 8: Triage Panels
When we extracted ACH processing off the monolith, the first architectural
question was where to run the new services. ECS Fargate was the obvious
choice for the worker fleet: persistent containers running queue consumers
with long-lived database connections and retry logic. But for the settlement
webhook receiver, we had a different problem. The question wasn't really
"Lambda or ECS?" It was whether the shape of the work was right for a single
compute model.
The work had two shapes. The settlement batch arrives from ProfitStars in a
burst: dozens to hundreds of callbacks in a short window as the overnight
ACH batch settles. The worker fleet, by contrast, needed sustained
throughput: consuming from SQS, running state machine transitions, calling the
allocator, writing to the database. Burst-on-demand and sustained-queue-
processing are not the same problem. Trying to solve them with one compute
model meant optimizing one and accepting compromises on the other.
The webhook receipt problem
When ProfitStars delivers a settlement callback, it expects a 200 response
within a few seconds. If it doesn't get one, it retries. ProfitStars isn't
doing you a favor when it retries; it's doing what it's supposed to do, and
you have to handle those retries safely. (We'd built idempotency keys at the
processor boundary for exactly this reason; see Part 4.)
On the original monolith, the webhook controller ran inline: receive the
callback, drive the state machine, run the allocator, write the ledger, return
- That worked fine until ACH volume grew and the settlement batch started arriving while other traffic was competing for the same Puma process pool. The callback endpoint would occasionally time out under load. ProfitStars would retry. The retries were safe, idempotent by design, but the load pattern was wrong for a shared host.
On an extracted ECS architecture with the webhook receiver also running inside
ECS, the same problem reappears in a different form: ECS worker throughput
becomes the bottleneck for whether ProfitStars gets a 200 or times out and
retries. If the worker fleet is saturated when the settlement burst arrives,
the receiver is stuck behind the workers.
Lambda at the inbound boundary
The solution was to keep receipt and processing structurally separate. Lambda
handled the webhook receiver: stateless, short-duration, and scales
immediately to any callback volume without pre-warming. The function received
the ProfitStars callback, validated the signature, enqueued the message to
SQS, and returned 200. That was the entire job.
The critical point is where in that sequence the 200 goes out. Lambda returns
200 to ProfitStars after enqueueing, not after processing. ProfitStars's
view of the system is just the webhook endpoint: is it responding? Lambda
answers that question cheaply and immediately, regardless of how many ECS
tasks are currently running or how saturated the worker fleet is.
This decoupling is what made the architecture resilient to settlement bursts.
ProfitStars doesn't care how long it takes to settle a payment. It cares
whether its callback got a 200.
Why not Lambda for the workers too?
The natural follow-up is whether Lambda could handle the worker side. It
couldn't, and the reasons are specific to what the workers actually do.
The ECS workers maintained persistent database connection pools. Processing
each payment involved multiple reads and writes: loading the payment record,
verifying the state transition was valid, running the allocator across
outstanding invoices, and writing ledger entries. Lambda's execution model
forces a new function invocation per message with no guarantee of connection
reuse. Managing a connection pool across Lambda invocations at settlement batch
volume was the wrong tool for the job.
Lambda's 15-minute timeout was also a mismatch. ProfitStars ACH submissions
involved retry logic with exponential backoff: a 5xx response from
ProfitStars could mean multiple retry cycles before a final outcome. We'd
sized the SQS visibility timeout above P99 ProfitStars response time to
prevent double-pickup on in-flight messages (more on that in Part 6). Lambda
can't hold a job open through that kind of retry window reliably.
ECS workers also supported true scale-to-zero. The settlement batch ran
overnight. Outside that window, we didn't need worker capacity. ECS Fargate
tasks could stop entirely until SQS depth rose again. Lambda has cold start
overhead that's tolerable for a short webhook receiver; it's less clean for a
sustained queue consumer that needs to start reliably at the beginning of a
predictable batch window every morning.
Back-pressure mechanics
When ECS workers are saturated and a new Lambda webhook fires (say, a second
settlement wave arrives while the previous batch is still being processed),
the sequence is: Lambda receives the callback, enqueues to SQS, returns 200 to
ProfitStars immediately. The message waits in the queue.
As the queue depth grows, the first alarm fires: ApproximateNumberOfMessages-
Visible above threshold for more than five minutes. That signals the worker
fleet isn't keeping up. ECS auto-scaling triggers additional tasks, but task
startup has latency. During that gap, if an individual message has been waiting
long enough, a second alarm fires: ApproximateAgeOfOldestMessage above 30
minutes.
Neither condition involves ProfitStars. ProfitStars got its 200 and closed
the connection. The back-pressure is entirely internal to our queue depth. The
queue absorbs the burst; the alarms tell us the workers need help; the workers
scale up. Lambda's decoupling from ECS is what made that flow possible.
Without it, ECS throughput directly determines whether ProfitStars times out
and retries.
Where the cost reduction actually came from
Extracting ACH workers off the monolith reduced infrastructure costs by
roughly 30%, but the savings came from two distinct sources.
The first was web tier right-sizing. Before extraction, Puma and Sidekiq
workers shared the same hosts. Those instances were sized to handle web
traffic plus worst-case ACH settlement batch simultaneously: a profile
driven by the peak ACH workload, not the actual web request shape. Once ACH
workers ran on ECS, the web tier could be sized to its actual web traffic
footprint. That was the larger share of the savings.
The second was ECS auto-scaling. ACH batch settlement happens overnight.
Outside the T+1 settlement window, ECS workers scaled to zero. The old Sidekiq
workers ran 24/7 on always-on instances regardless of whether ACH was actively
processing; Sidekiq's process model doesn't support true scale-to-zero
cleanly. ECS Fargate tasks can stop entirely between batches and restart when
SQS depth rises again. That's a fundamentally different cost curve.
Both effects were real. The web tier right-sizing produced the larger number;
the scale-to-zero was the cleaner outcome once we stopped paying for idle ACH
workers during business hours.
What's next
The Lambda-at-boundary pattern made the webhook receipt problem disappear. The
SQS queue between Lambda and ECS absorbed the back-pressure without involving
ProfitStars at all. But the queue itself had design decisions that were not
obvious: FIFO vs standard, visibility timeout sizing, MessageGroupId strategy
for per-payment ordering, and the idempotency key that prevented duplicate
state transitions when a worker double-picked a message.
Part 6 covers the SQS configuration decisions: specifically the one that bit
us first and the key that saved us twice.
Top comments (0)