Series — Building ACH on AWS: 0: System Design · 1: Sidekiq Latency · 2: Extraction Boundary · 3: Shadow Traffic · 4: What Broke · 5: Lambda/ECS Split · 6: SQS FIFO · 7: Instrumentation · 8: Triage Panels
We had already decided to extract ACH processing off the monolith. That decision was forced on us: Sidekiq ACH workers and Puma web workers were competing for shared host resources during T+1 settlement windows, and web latency was degrading in a predictable, infuriating pattern every morning. The extraction was going to happen.
What took a week to settle was where to cut.
The maximalist option we almost shipped
The natural reading of "extract ACH" is: move everything payment-shaped out of the monolith. ProfitStars HTTP calls, Sidekiq job execution, settlement callback receipt, the state machine, the allocator, and the ledger writes. Build an autonomous payment service that owns its own state and its own data. The monolith becomes a client of that service.
This is the strangler-fig pattern taken to its logical extreme. And it looks clean on a whiteboard. One service. Clear ownership. The monolith doesn't know anything about payments.
We considered it seriously. We rejected it for three specific reasons, and I think those reasons generalize: they show up in almost every monolith extraction decision I've seen since.
Trap 1: the distributed transactions disguise
The allocator writes to two tables: invoice_allocations and payment_ledger_entries. Both tables are owned by the monolith's billing and invoicing domain. They're used by the invoice balance calculations, the IOLTA routing logic, and the firm's financial reporting. Nothing about those tables belongs to a payment processing service: they're billing tables that happen to be updated when a payment settles.
If you move the allocator to an external service, you have a service that doesn't own the tables it writes to. Every allocation becomes a cross-service write: the external service making database calls into the monolith's schema, or the monolith exposing a write API that the external service calls as part of its settlement flow.
That second option sounds more correct (the external service calls an API), but it just reframes the problem. Now you have a multi-step operation that spans a network boundary: the external service needs to (1) update ProfitStars's response into payment state, and (2) trigger allocation writes in the monolith. If the first succeeds and the network fails before the second, you have a confirmed payment with no ledger entries. That's not an edge case. That's a data consistency bug with compliance implications for a trust ledger.
Keeping the allocator in the monolith meant the whole settlement path (state transition and ledger writes) could happen in a single ACID transaction. Moving it out would have required us to implement a saga or two-phase commit to recover the same guarantee across a network boundary. That's coordination infrastructure we'd have to build, maintain, and debug. The monolith was already correct. We didn't want to replace a working transaction with a distributed protocol.
Trap 2: hidden UI consumers
The state machine methods weren't just called from the async processing path. They were called by Rails controllers to determine what to show the user. Invoice lock status (whether an invoice is locked against new payment attempts because an ACH submission is in flight) is determined by reading payment state. The invoice show page, the payments dashboard, the checkout pre-flight check: all of them called into the state machine.
If the state machine lives in an external service, every one of those UI code paths becomes a remote call. The checkout controller that checks whether an invoice is already locked before accepting a new payment now has to wait for a network response. The invoice show page now has a new failure mode: it can't render correctly if the payment service is slow or down.
The async processing path (submission job, callback handler) was the right thing to move. Those are batch flows that run outside the request cycle. A 200ms latency increase in the settlement worker doesn't affect web performance. But the UI consumers are in the request path. Moving the state machine to a remote service would have made every invoice page slower and more fragile, in exchange for no architectural benefit.
Trap 3: the idempotency guarantee doesn't live in the runner
Part 1 of this series describes how we built the idempotency key: a UUID minted at payment record creation, stable across retries, stored on the record so it could never be regenerated from inputs. That guarantee lives at the processor boundary. ProfitStars gets the UUID; if it sees it again, it returns the original result.
The extracted service's job is to honor that guarantee: to carry the UUID from the payment record to ProfitStars and back, handling retries correctly. That's a pure I/O responsibility. The service knows how to talk to ProfitStars. It does not need to know what a settlement response means.
What a settlement response means (which state transitions are valid, which invoices get allocated, how the ledger entries get tagged) is business logic. It depends on the invoice data, the IOLTA routing rules, the partial-payment ordering. All of that context lives in the monolith. The extracted service calling back into the monolith to trigger state transitions isn't an architectural compromise. It's the correct separation: one side owns I/O mechanics, the other side owns business meaning.
The boundary we drew
The checkout controller publishes to SQS instead of enqueuing a Sidekiq job. ECS workers consume from that queue, call ProfitStars, and return; they write nothing. On settlement, ProfitStars POSTs to a Lambda function that validates the signature, enqueues the event to a second SQS queue, and returns 200 immediately. ECS settlement workers consume that queue and call the monolith's internal API to trigger the state transition. The monolith's state machine fires, the allocator runs, and invoice_allocations and payment_ledger_entries are written in the same database transaction they always were.
The extracted service is stateless with respect to business logic. It doesn't know what "confirmed" means. The monolith is stateless with respect to processor mechanics. It doesn't retry ProfitStars HTTP calls. The boundary is clean because it separates two different kinds of responsibility, not just two deployment units.
What this cost us
Keeping the state machine and allocator in the monolith meant the ECS workers needed to call back into the monolith to finish a settlement. That callback is a synchronous HTTP call in the critical path of settlement processing. We had to design it carefully: make it idempotent, give it a tight timeout, and make sure the ECS worker retried correctly if the callback failed.
That's real complexity. The alternative (cross-service table writes or distributed transactions) was worse complexity. We chose the boundary that kept the hard problem (data consistency) in the place that already solved it (the monolith's ACID layer), and accepted the simpler overhead of a callback API.
Part 3 covers how we ran the old Sidekiq path and the new SQS/ECS path in parallel before cutting over, and why the shadow traffic comparison required more thought than "run both and check if the outputs match."
Top comments (0)