Series — Building ACH on AWS: 0: System Design · 1: Sidekiq Latency · 2: Extraction Boundary · 3: Shadow Traffic · 4: What Broke · 5: Lambda/ECS Split · 6: SQS FIFO · 7: Instrumentation · 8: Triage Panels
The ledger post in this series (Part 1) established the first layer of
protection: a UUID idempotency key minted at payment record creation,
re-submitted to ProfitStars unchanged on every retry of the same
in-flight charge. ProfitStars returns the result of the original attempt
rather than process a second charge. That layer lives at the application-
to-processor HTTP boundary.
The SQS queue design added a second, distinct layer at a different
boundary: processor callback to our Lambda receiver. The two layers guard
against different failure modes. Understanding why both were necessary is
what this post covers.
The ordering problem with a standard queue
SQS standard queues do not guarantee message ordering. For most
consumers that tradeoff is fine. For a payment state machine it is not.
ProfitStars ACH submissions generate a sequence of callbacks: submitted
(the ACH batch was accepted), followed by settled (funds moved) or
returned (the ACH was rejected). These arrive async, often minutes or
hours apart. Our ECS workers consumed them and drove the payment state
machine forward. Valid transitions went one direction only: pending to
submitted, submitted to confirmed or failed. There was no valid
path from pending directly to confirmed.
If a settled callback was processed before the submitted callback for
the same payment (which a standard queue permits), the state machine
would attempt an invalid transition. Depending on error handling, it
either threw and sent the payment into a retry loop, or silently dropped
the settled event.
We used a FIFO queue. The ordering guarantee was not optional when events
were driving state transitions that routed money into trust accounts.
MessageGroupId = payment_id
Within a FIFO queue, SQS still processes messages concurrently across
different message groups. We set MessageGroupId to payment_id. Every
message for the same payment was in the same group, delivered in
insertion order. Messages for different payments processed concurrently.
One payment processed correctly; many payments processed fast.
DeduplicationId = processor_event_id
ProfitStars assigns a unique event ID to each callback it fires. We used
that as the SQS DeduplicationId.
This is the second idempotency layer, and it is different in kind from
the UUID at the API boundary. The UUID layer guarded against a submission
retry from our side: we called ProfitStars, timed out, called again,
and needed to prevent a second charge. The DeduplicationId layer
guarded against ProfitStars calling us more than once for the same event.
Processors have their own callback retry policies. If Lambda returned a
200 slowly or ProfitStars had a delivery failure, we could receive the
same settled event twice. SQS FIFO's deduplication window is five
minutes: the same DeduplicationId arriving again within that window is
dropped before it reaches a worker.
The message schema was: payment_id, event_type (submitted,
settled, failed, or returned), processor_event_id, amount,
timestamp. The processor_event_id appeared twice: once in the
message body (for logging and alerting on deduplication hits) and once as
the DeduplicationId on the envelope (for SQS to act on).
Visibility timeout sizing
We sized the visibility timeout at P99 ProfitStars response time
multiplied by two. Not average, and not P90.
The reason is in Part 4, which covers what broke during extraction. The
first failure was a visibility timeout set too short. ECS workers calling
ProfitStars occasionally exceeded the timeout (not on typical requests,
but on the tail). SQS re-enqueued the message. A second worker picked it
up and submitted a duplicate ACH. The UUID idempotency key caught the
duplicate at the processor boundary, so no double charge occurred, but
it produced confusing duplicate payment records and inflated retry counts
in Datadog before we identified the misconfiguration.
The visibility timeout must cover your actual worst case, not your
typical case. A timeout at P90 will fail on the worst 10% of requests.
For a payment state machine that is an unacceptable failure rate. We
moved to P99 x 2, accepted the slight increase in redelivery latency on
genuine worker crashes, and eliminated the false-duplicate problem.
Retry and backoff
When a worker called ProfitStars and received a response, we handled it
differently depending on the response class.
A 5xx response meant ProfitStars had a problem on their side: exponential
backoff with jitter, 1 second base, 30 second maximum, five retries.
Jitter matters here because T+1 settlement batches hit all workers
simultaneously. Without it, every worker that got a 5xx during a batch
would retry at the same intervals, creating synchronized bursts against
an already-struggling processor.
A 4xx response was immediate failure. We transitioned the payment to
failed, fired an alert, and did not retry. A 4xx means we sent bad data
or the request was unauthorized. Retrying the same malformed request
produces the same 4xx. Retrying would only delay the payment reaching a
state where someone could investigate it. A 5xx says nothing about
whether our request was valid; a 4xx says it was not.
A network timeout re-submitted with the same payment UUID. The payment
stayed in its current state and re-entered the retry loop. ProfitStars
returns the original result for a known idempotency key, so the retry was
safe by construction.
The DLQ and what manual review actually means
After five failed retry attempts, SQS moved the message to a dead letter
queue. We had a Datadog alert wired to DLQ depth > 0 at high severity.
"Manual review" in the runbook meant something specific. A DLQ message
had exhausted all automated recovery. Someone needed to open that
payment_id, read the error history, and determine what happened. Was
ProfitStars in an extended outage? A data problem specific to this
payment? A bug from a recent deploy that only surfaced on this edge case?
The DLQ was not a retry buffer. It was a stop: here is a payment we
could not process automatically; go look at it.
The high-severity classification for DLQ depth (versus lower-severity
for general queue backlog) reflected that distinction. A slow queue might
self-resolve. A DLQ message had been through five retries spread across
minutes of backoff. It was not going to self-resolve.
Two layers, two failure modes
The UUID at the ProfitStars API boundary stopped us from accidentally
charging a client twice when our submission retried after a timeout.
Without it, a timed-out request followed by a retry produces two processor
charges for the same payment record.
The DeduplicationId at the SQS boundary stopped ProfitStars from
triggering two state machine transitions when their callback delivery
retried. Without it, a duplicate settled event reaching the ECS worker
would attempt to transition an already-confirmed payment to confirmed
again, an error or silent corruption of the payment's event log.
Neither layer was redundant. Each guarded a boundary where different
things could go wrong. The idempotency key at the API boundary was a
defense against our own retries. The DeduplicationId at the queue
boundary was a defense against the processor's retries. Together they
meant that even in the presence of network failures and callback
redelivery, each payment advanced through the state machine exactly once.
Part 7 covers instrumentation: how we measured state machine
transitions, detected deduplication hits, and sized our SQS depth alerts
to page on the right things without generating noise during the
predictable T+1 settlement batch window.
Top comments (0)