Series — Building ACH on AWS: 0: System Design · 1: Sidekiq Latency · 2: Extraction Boundary · 3: Shadow Traffic · 4: What Broke · 5: Lambda/ECS Split · 6: SQS FIFO · 7: Instrumentation · 8: Triage Panels
The failure that surprised me most during the ACH extraction wasn't the one I was
watching for.
We had moved ACH submission off Sidekiq and onto SQS-backed ECS workers. We had a
feature flag controlling which path new submissions took. We had run two weeks of
shadow traffic and a load test against the ProfitStars sandbox. The cutover looked
clean. Then we started seeing state machine conflicts.
A payment would enter submitted via the new ECS path. ProfitStars would later POST
the settlement callback to the same endpoint it always had, the monolith webhook
controller. That controller would advance the payment to confirmed using the old
Sidekiq code path. If the ECS worker had already picked up the settlement event from
SQS (routed through the Lambda receiver we had wired up), both paths were now
advancing the same payment's state machine simultaneously. The transition from
submitted to confirmed was firing twice, from two different processes, against
the same payment record.
That was the second thing to break. Here's all three, in the order we found them.
1. SQS visibility timeout too short
The first sign of trouble was confusing rather than alarming. Datadog was showing
elevated retry counts for ACH submission jobs, and a handful of payments had two
records in the submissions table for the same payment ID.
The idempotency key held. When the second ECS worker submitted a payment that was
already in-flight, ProfitStars returned the result of the original attempt rather
than processing a duplicate (that's what the idempotency key is for, and it worked
as designed). No customer was charged twice. But the duplicate records were real, the
retry counts in Datadog were inflated, and neither reflected an actual failure.
The root cause was the SQS visibility timeout. We had set it to 30 seconds,
roughly the average ProfitStars response time during our load test. The problem is
that average is the wrong number. ProfitStars P99 response time during T+1 settlement
batch windows ran longer. When a worker hit one of those tail-latency responses and
exceeded the visibility timeout, SQS assumed the message had failed and made it
visible again. A second worker picked it up and submitted the same payment.
The fix was to set the visibility timeout above P99 ProfitStars response time, not
P50 and not a round number we had guessed at. We pulled the actual P99 from our load
test results and set the timeout to P99 times two as a margin, the same formula we
used for retry backoff configuration.
The visibility timeout is not "how long the job takes on average." It is "how long
the job can take in the worst case you are willing to tolerate before treating it as
stuck." Set it to the average and half your tail-latency jobs produce phantom retries.
The idempotency key saved us from a real duplicate charge; the noise in Datadog is
how we found the misconfiguration at all.
2. Callback URL routing during feature-flag cutover
This was the split-brain problem. The feature flag controlled which path ACH
submissions took: monolith Sidekiq or SQS/ECS. But the flag only covered
submissions. ProfitStars settlement callbacks had been POSTing to the same monolith
webhook endpoint for years, and we had not changed where they went.
During the partial rollout (10% of new submissions routed through ECS, 90% still
through Sidekiq), the routing split was invisible from ProfitStars's perspective. It
submits a payment; it settles; it POSTs the callback to the URL it has on file. The
callback always went to the monolith.
For the 90% still on Sidekiq, that was correct. Callback hits the monolith, Sidekiq
advances the state machine to confirmed, allocator runs, ledger updates.
For the 10% running through ECS, the Lambda webhook receiver was also listening for
settlement events (by design). When ProfitStars POSTed the settlement callback, the
monolith webhook controller picked it up AND the Lambda receiver enqueued the same
event to SQS for the ECS workers. Both paths tried to advance the same payment from
submitted to confirmed. The state machine had a guard at the Rails level but not
a distributed one; two processes racing on the same payment record produced
conflicts. In some cases confirmed fired twice. In others, one transition lost the
race and left the payment in a partially-advanced state that required manual
correction.
The fix was to make the feature flag gate callbacks and submissions together. During
the incident we configured ProfitStars to update its callback URL in step with the
rollout percentage, coupling both routing decisions. The cleaner solution in
retrospect was to route all ACH callbacks through Lambda from day one of the
extraction, with the monolith webhook controller delegating to it rather than
handling ACH directly.
A feature flag that covers only half the data flow creates a split-brain during the
rollout window. "Submission routing" and "callback routing" are one round-trip from
ProfitStars's perspective, even if they are separate endpoints in your application.
3. DB connection pool exhaustion on ECS tasks
The third failure looked like a ProfitStars outage. The SQS queue depth was climbing.
ECS workers were processing slowly or not at all. We had seen ProfitStars have slow
periods before. The first instinct was to check their status page.
The status page was clean. ECS worker logs showed workers not failing but waiting.
DB connection wait time in the Rails connection pool metrics showed a queue building
up; workers were trying to acquire a database connection, could not get one, and
sitting idle. SQS depth grew because workers were not processing. It looked like an
external dependency problem until we found the right dashboard.
The ECS task configuration had been copied from the monolith with one adjustment: we
had right-sized the CPU and memory allocation for queue workers rather than web
servers. But we had left the Rails database connection pool at the same size the
monolith used, a pool sized for Puma web request concurrency, where you need enough
connections to serve simultaneous HTTP requests across multiple threads.
ECS settlement workers do not serve HTTP requests. Each task pulls from SQS and
processes one message at a time. The concurrency model is: how many tasks are running
in parallel, not how many threads per process are handling requests. When the T+1
settlement batch arrived and all ECS workers became active simultaneously, the total
connection demand across all running tasks (each holding a pool sized for web
concurrency) exceeded what RDS was configured to accept.
The fix was to size the ECS task connection pool to match queue worker concurrency:
small, because a queue worker does not need multiple idle connections per task waiting
for web traffic that will never come. Once we resized and redeployed, the next
morning's batch cleared without incident.
A Puma web server and a queue worker share a Rails runtime but not a concurrency
model. The configuration that works in a monolith does not automatically carry over
to an extracted service. Moving a config value without questioning the assumption it
encodes is how you get a failure that looks like an external outage.
The pattern underneath all three
None of these failures were inside the monolith. None were inside the new
ECS/Lambda/SQS stack. All three were at the boundary: in assumptions that migrated
across without being questioned.
The visibility timeout was sized for average load test behavior, not worst-case
production behavior. The callback routing assumed "submission path" and "callback
path" were governed by the same flag, because they always had been in the monolith.
The connection pool assumed "Rails service" meant "web concurrency model," because
the monolith was a web server.
Migration bugs live at the seam. The old system encodes its assumptions in config
values, routing conventions, and behaviors that were correct in their original context
and weren't designed to survive extraction. When something breaks after a migration,
the places to look aren't the parts you built; they're the parts you moved.
Next: Part 5. Lambda for Webhooks, ECS for Workers: Splitting a Payment Service.
Why the stateless webhook receiver and the stateful queue workers need different
compute primitives, and what the boundary between them costs you when you get it
wrong.
Top comments (0)