Series — Building ACH on AWS: 0: System Design · 1: Sidekiq Latency · 2: Extraction Boundary · 3: Shadow Traffic · 4: What Broke · 5: Lambda/ECS Split · 6: SQS FIFO · 7: Instrumentation · 8: Triage Panels
When a law firm collected a client payment through MyCase, the Rails monolith
handled everything. The checkout controller created the payment record, Sidekiq
submitted it to ProfitStars, ProfitStars called back with the settlement result,
and the monolith wrote the ledger entries. All of that ran inside a single
application, sharing process-pool resources with every web request in the
system.
This post is the architecture overview for a series about what we changed and
why. It's not the story of the extraction; that starts in Part 1. It's a map
of the system: what the monolith handled, what moved out, where the boundaries
are, and one design constraint that shaped every other decision.
The monolith before extraction
ACH on MyCase meant electronic bank debits. A client opens a payment link,
enters bank account details, and authorizes a debit. MyCase submits the
transaction to ProfitStars, the ACH processor. ProfitStars initiates the debit
through the ACH network. Settlement arrives at T+1 or T+2 (the next or
second business day) when ProfitStars confirms the funds transferred.
Before extraction, that entire flow ran in the monolith:
The problem with this design wasn't correctness; it worked. The problem was
that Sidekiq workers and Puma web workers ran on the same hosts and competed
for the same process pool. ACH settlement arrives in batches at T+1, typically
around 8am. When the overnight batch landed, Sidekiq workers consumed available
processes. Web request latency climbed. The two workloads had different
resource needs and no isolation between them.
What the payment state machine did
The state machine was the heart of the system. Every payment moved through a
defined set of states with explicit transitions. Not every state is shown here,
but the main ACH path:
pending was created at checkout. submitted happened after Sidekiq sent the
payment to ProfitStars and received an acknowledgment. confirmed happened after
ProfitStars sent the T+1 settlement callback. failed or returned happened if
ProfitStars rejected the payment or the bank returned the debit.
The allocator ran after confirmed. It applied the payment amount against the
client's outstanding invoices and wrote ledger entries inside a single database
transaction. For IOLTA accounts (trust accounts where legal regulations govern
how funds are tracked) the allocator had to route the entries correctly before
the payment was considered fully processed.
What moved out of the monolith
The extraction moved ProfitStars communication and async queue processing to
a separate service layer on AWS. The state machine and allocator stayed in the
monolith. That boundary was deliberate: moving the state machine out would
have required distributed transactions to coordinate writes across two data
stores, and the allocator had implicit dependencies on Rails models that the
extraction would have had to replicate.
The extracted architecture:
The shape of the message at each queue boundary is worth noting. The SQS
submission message contained payment_id and an idempotency key. The
settlement message contained the ProfitStars processor_event_id (used as
the FIFO deduplication ID), the payment_id, and the settlement status. Neither
message contained payment credentials.
The PCI constraint that shaped the data flow
MyCase did not store bank account numbers, routing numbers, or card numbers.
This was a deliberate decision to reduce PCI DSS scope. Storing raw payment
credentials would have required MyCase to meet PCI compliance requirements for
data at rest: encryption, key management, access controls, audit trails.
Instead, when a client entered bank account details at checkout, ProfitStars
handled the credential capture and storage. ProfitStars issued a payment method
token in return. That token is what MyCase stored in the payments database.
The token is opaque to MyCase. It is a reference that ProfitStars can resolve
to the underlying bank account or card on file. MyCase never has the raw
credentials; ProfitStars holds them in its own PCI-compliant infrastructure.
This meant the ECS submission worker's "read payment record from DB" step
retrieved a payment method token, not bank credentials. The worker passed the
token to ProfitStars along with the payment amount and idempotency key. ProfitStars
resolved the token and initiated the ACH debit. The ECS worker knew the amount
and the token reference; it never knew the account number.
The same pattern applied to card payments, which ran on a parallel track through
WorldPay. Token in the DB, raw card data held by the processor.
The monolith internal API
The ECS workers needed to trigger state machine transitions in the monolith after
processing events. Rather than writing directly to the database, which would
bypass the state machine guards and the allocator, the workers called a
VPC-private internal API that the monolith exposed.
The endpoint accepted an HTTP POST. It was not exposed to the public internet;
it was accessible only within the VPC. Authentication used a shared secret passed
in a request header. The ECS task's IAM role did not grant direct DB access for
state machine writes; it called the internal API, which ran the state machine
transition and the allocator inside the monolith's own application context.
This preserved the invariant that every state transition and every allocator run
happened through the same Rails code path, with the same guards, regardless of
whether the trigger came from the web tier or the ECS worker fleet.
The SQS FIFO design
The queues were SQS FIFO, not standard.
Standard SQS provides at-least-once delivery and best-effort ordering. For
payment events, both properties were wrong. A payment submitted twice would
charge a client twice. A settlement callback processed before submission would
cause a state machine conflict.
FIFO queues provided exactly-once delivery within a deduplication window and
strict ordering within a message group. The submission queue used payment_id
as the MessageGroupId, ensuring that retry events for the same payment were
processed in order by the same consumer group. The settlement queue used
processor_event_id as the DeduplicationId, so duplicate callbacks from
ProfitStars (which the ACH network can generate) were automatically rejected at
the queue layer rather than reaching the state machine.
The Lambda boundary
ProfitStars expected a 200 response to webhook POSTs within a few seconds. The
monolith's original webhook controller returned 200 after running state machine
logic; the response time depended on Rails application performance under T+1
batch load. If the monolith was under pressure when the settlement batch arrived,
slow responses risked ProfitStars treating the callback as failed and retrying.
Lambda decoupled the acknowledgment from the processing. The webhook receiver
validated the request signature, published the event to SQS, and returned 200,
all in under 100ms. The ECS settlement workers processed from the queue on their
own schedule. ProfitStars received its acknowledgment immediately; the monolith
processed the event when its workers had capacity.
What the series covers
The eight posts that follow this one each cover a specific problem in the
extraction. They assume the architecture above as context.
Part 1: Why the extraction started at all: Sidekiq and Puma competing for
process-pool resources during T+1 settlement batches, and how web latency
became the trigger.
Part 2: Where to draw the extraction boundary: what the ECS/Lambda layer
handles, what stayed in the monolith, and why moving the state machine out would
have been the wrong call.
Part 3: The shadow traffic migration: running both stacks in parallel,
the CircleCI divergence gate, and the percentage rollout before full cutover.
Part 4: Three things that broke: SQS visibility timeout misconfiguration,
callback routing split-brain during partial rollout, and DB connection pool
exhaustion.
Part 5: Lambda for webhooks, ECS for workers: why the acknowledgment
boundary and the processing boundary needed different compute primitives.
Part 6: SQS FIFO in detail: message group design, deduplication, visibility
timeout sizing, and DLQ as the high-severity signal.
Part 7: Instrumenting the state machine: which metrics to add, which to
reject, and the three noise sources that generated 35% of false-positive alerts.
Part 8: Triage at 8am: the two-panel dashboard split and the runbook that
made on-call routing take two minutes instead of twenty.
The series assumes familiarity with Rails, SQS, and Lambda at a conceptual
level. Posts do not re-explain the state machine, allocator, IOLTA routing, or
basic AWS service primitives. This post is the one to return to if any post
references "the monolith internal API" or "the payment method token" and the
context is unclear.
Top comments (0)