Series — Building ACH on AWS: 0: System Design · 1: Sidekiq Latency · 2: Extraction Boundary · 3: Shadow Traffic · 4: What Broke · 5: Lambda/ECS Split · 6: SQS FIFO · 7: Instrumentation · 8: Triage Panels
The most common way to migrate live payment processing is to find a
maintenance window, take the system down, deploy the new path, and hope the
tests were good enough. We could not do that. ProfitStars ACH callbacks
arrive at T+1 and T+2: any maintenance window long enough to matter would
either drop settlement callbacks mid-flight or require us to negotiate
blackout windows with the payment processor. Neither was on the table. The
new ECS-based path had to reach production quality while the existing Sidekiq
path kept processing real money.
The answer was to run both stacks in parallel for roughly two weeks, compare
their outputs on every payment that passed through, and make the CircleCI
gate a hard block before any cutover deploy could ship.
Where we left off
Part 2 covered where we drew the extraction boundary: ProfitStars
communication and async job execution moved to ECS; the state machine and
allocator stayed in the monolith. That boundary is what made the shadow
period possible. The ECS path was stateless with respect to business logic:
it communicated with ProfitStars and reported outcomes, but data writes stayed
in the monolith's hands. There was a clear seam to compare across.
Two paths, one authoritative
During the shadow period, every new ACH submission went through both paths.
The Sidekiq path was authoritative: it finalized state transitions, ran the
allocator, and wrote ledger entries. The ECS path ran concurrently, processed
the same payment, and reported its outcome without committing anything.
The ECS path ran in shadow mode: it received the same inputs as
the Sidekiq path but did not commit state transitions. We compared the
ProfitStars responses and the resulting state machine outcomes from both
paths. Divergence meant the two paths produced different final states or
different processor responses for the same payment. The idempotency key was
the protection that made this safe: the ECS shadow path used the same payment
UUID that the Sidekiq path had already submitted, so if the ECS path
touched ProfitStars, the processor returned the result of the original
submission rather than processing a second charge.
The CircleCI gate
We encoded the divergence requirement as a CircleCI quality gate. The
pipeline could not produce a cutover deploy artifact if shadow comparison
results showed divergence above 0.1% (fewer than one in a thousand payments
where the two paths produced different outcomes). This was not a soft
guideline or a dashboard metric the team agreed to eyeball before shipping.
It was a hard pipeline block. If the gate failed, the deploy did not happen.
The gate ran on a rolling window of shadow comparisons collected from live
production traffic. Alongside it, we required a load test step: before the
cutover gate could pass, the ECS worker fleet had to survive a test against
the ProfitStars sandbox at 2x production volume. The sandbox test validated
fleet sizing for the T+1 settlement batch peak (not steady-state traffic,
but the worst-case burst that arrived every morning when overnight ACH settled).
We also wrote RSpec tests that replayed webhook fixture sequences against
both paths: the full set of ProfitStars callback shapes we had seen in
production, including happy path, partial payment sequences, returned-payment
callbacks, and the timeout-then-settle sequence. The goal was to verify that
the allocator produced identical ledger entries regardless of which path drove
the state transitions. These ran in CI and had to stay green throughout the
shadow period.
Why the percentage rollout, not 0% to 100%
Once the shadow gate passed, we did not flip all traffic at once. We routed
new ACH submissions through ECS in increasing increments via a feature flag:
10% first, then 50%, then 100% after each step held stable for 24 hours.
The reasoning was about blast radius. The shadow period compared outcomes on
every payment but did not commit state from the ECS path; the Sidekiq path
had been authoritative the whole time. The 10% live increment was the first
moment ECS was actually responsible for real state transitions on real money.
If something broke, 10% of ACH submissions were affected and we could roll
back by toggling the flag. A 0%-to-100% jump would have turned a potential
10% incident into a full one.
The 50% step served a different purpose. We had load-tested at 2x production
volume against sandbox, but there is a difference between a synthetic load
test and the actual shape of live traffic: payment retries, partial
settlements, concurrent callbacks for the same payment arriving at T+1 batch
time. Fifty percent was the first time the ECS fleet was absorbing real
production load at a meaningful scale. We watched Datadog for worker
saturation, SQS depth, and ECS task health for 24 hours before moving to
100%.
What the two-week window cost
Running two paths in parallel had real overhead. The Sidekiq infrastructure
stayed live and had to be kept healthy. Bug fixes to payment submission logic
had to be applied to both paths simultaneously. Shadow comparison noise
(payments where the two paths differed for benign reasons like timing
differences in observing ProfitStars response ordering) had to be
investigated and either whitelisted or fixed before the gate could pass.
That overhead was the point. The shadow period was not a grace period before
the real work started. It was the production validation. Every day the
comparison stayed below 0.1% divergence was a data point that the ECS path
was behaviorally equivalent to the one that had been processing real money
for years. The CircleCI gate made that evidence mandatory, not optional.
The cutover
The gate passed on day fourteen. The feature flag moved to 100%. The Sidekiq
ACH workers were turned off. The migration was done.
Then, over the following week, three things broke in ways the shadow period
had not caught. Part 4 covers what broke first, starting with why the SQS
visibility timeout was configured wrong and what happened to in-flight
payments before anyone noticed.
Top comments (0)