Series — Building ACH on AWS: 0: System Design · 1: Sidekiq Latency · 2: Extraction Boundary · 3: Shadow Traffic · 4: What Broke · 5: Lambda/ECS Split · 6: SQS FIFO · 7: Instrumentation · 8: Triage Panels
The alert fired at 8:15am on a morning when T+1 settlement was already due.
SQS oldest-message threshold crossed. The first instinct on the team was
ProfitStars: the settlement batch was scheduled, the timing lined up,
and ProfitStars outages during batch windows had happened before. I opened
the Datadog dashboard and went straight to the external error rate panel.
Flat. ProfitStars HTTP client was quiet.
Then I checked the internal Rails exception rate panel. It had been spiking
since 8:02am (thirteen minutes before the SQS alert fired). The queue had
been absorbing the impact. Messages were retrying, visibility timeouts were
doing their job, and the depth stayed below the threshold while the worker
fleet chewed through the retry backlog. By the time the oldest-message alarm
went off, the DLQ already had entries.
The root cause was a nil-reference error in the state machine handler for
returned ACH payments (a status we hadn't exercised heavily in load
testing). The code path existed, the state was valid, but one edge case
around the return reason code produced a nil dereference that the state
machine caught, retried, and eventually routed to the DLQ.
Without the two panels, the first move on any payment alert would have been
to open ProfitStars's status page. On that particular morning, that would
have been ten minutes of looking at a green status page while the actual
problem sat in Rollbar waiting to be found.
Why one alert metric is not enough
When ACH payments stop processing, three meaningfully different things could
be happening:
- ProfitStars outage: all submissions failing 5xx, DLQ not yet populated because messages are still in retry backoff.
- Internal Rails bug: specific payment IDs failing, DLQ populating, ProfitStars healthy, pattern traceable to a deploy or input shape.
- Capacity: queue backing up with no errors, ECS auto-scaling lag, the fix is worker count not code.
All three look identical at the SQS depth layer. Each has a completely
different first move. Combining them into a single alert means the on-call
engineer's first question is always "Is ProfitStars down?" regardless of
what's actually broken. Five to ten minutes wasted on every internal bug
page.
The dashboard design we landed on split the signal at the source layer.
The two panels
The Datadog dashboard for ACH payments had separate panels for:
External error rate: requests to the ProfitStars HTTP client, segmented
by response code category. A spike here meant ProfitStars was returning 5xx
or timing out. A flat line here ruled out the processor as the root cause.
Internal exception rate: Rails exceptions raised inside the ECS worker
fleet, specifically in the payment processing and state machine layers. A
spike here meant something in our code was throwing. Correlate with Rollbar
and recent CircleCI deploys.
The relationship between the two panels was the signal. External spikes first:
escalate to ProfitStars. Internal spikes first: page on-call to check code.
Both flat while queue depth grows: look at ECS worker count.
The thirteen-minute gap on that morning was legible because I could see which
panel moved first. The external panel's flatness wasn't ambiguous; it was a
data point. I wasn't guessing that ProfitStars was fine. The panel told me.
The runbook the panels supported
The two-panel split was only useful because the on-call runbook was structured
around it. An alert without a documented decision tree still produces
five-minute debates about where to look first. The runbook for "ACH payments
backing up" read as follows:
Step 1. Check Datadog SQS depth and oldest-message dashboards. Establish
whether the queue is growing, stable, or already draining.
Step 2. If DLQ depth is greater than zero, inspect the DLQ messages.
Read the error field. Is it a ProfitStars 5xx response, or a Rails exception
class with a stack trace?
Step 3. If the DLQ error is a ProfitStars response code: check the
ProfitStars status page. If confirmed outage, stop new ACH submissions via
the feature flag. Notify affected firms via the standard payment-interruption
template.
Step 4. If the DLQ error is a Rails exception: open Rollbar and pull the
stack trace. Cross-reference with the CircleCI deploy history for the ECS
worker service. The deploy timestamp and the exception spike timestamp should
bracket the window.
Step 5. If the queue is backing up with no errors in the DLQ and no
exception spike: open CloudWatch and check ECS worker task count against
the target. Auto-scaling lag is the likely explanation. Manually trigger a
scale-out if the backlog is growing faster than new tasks will resolve it.
The decision tree looks obvious written out. What made it non-obvious in
practice was that step 2 (inspecting the DLQ messages) required knowing
that the DLQ was the right place to look and not the ProfitStars status page.
The runbook made that explicit. The Datadog panels made step 2 unnecessary
in most cases because the external vs internal split was already visible
before you looked at individual messages.
The 35% alert noise reduction
The triage split was one piece of a larger instrumentation effort that cut
false-positive pages by about 35%. Three noise sources, three fixes, same
underlying principle: alerts should distinguish real problems from expected
behavior before they page anyone.
ECS auto-scaling events triggered spurious "payments unavailable" alerts during
task replacement. Fix: a three-minute evaluation window so a single scaling
spike wouldn't fire.
ProfitStars transient timeouts at the P90 response time (~8 seconds) were
alerting against a P50 threshold. Fix: reconfigure to P99 plus 20% buffer.
T+1 settlement batches arrived predictably around 8am and caused SQS depth
spikes that anomaly detection treated as incidents. Fix: scheduled maintenance
window with a higher threshold during the batch window.
What the series covered
This is the eighth and final post in the series. It's worth pausing to map
what the eight posts together traced.
Post 1 started at the trigger: Sidekiq ACH workers and Puma web workers on
shared hosts, competing for process-pool resources during T+1 batch windows,
degrading web latency.
Post 2 defined the extraction boundary: what moved out, what stayed in the
monolith, and why the state machine and allocator stayed where the data lived.
Post 3 covered the shadow traffic migration: parallel runs, a CircleCI gate
on divergence below 0.1%, and a feature-flag percentage rollout before full
cutover.
Post 4 cataloged three things that broke during extraction: SQS visibility
timeout misconfiguration, callback routing conflict during partial rollout,
and DB connection pool exhaustion that initially looked like a ProfitStars
outage.
Post 5 described the Lambda-at-boundary pattern: stateless webhook receiver
returning 200 to ProfitStars immediately, decoupling processor acknowledgment
from ECS worker availability.
Post 6 went deep on SQS FIFO: MessageGroupId per payment_id, DeduplicationId
from the processor event, visibility timeout above P99 ProfitStars response
time, and DLQ as the high-severity signal for exhausted retries.
Post 7 described state machine instrumentation: transition counters,
time-in-state histograms for settlement latency, and a pending-aging alert
on the allocator.
This post closed the operations picture: dashboard structure, runbook design,
and the triage discipline that makes the whole system operable at 8am when
the settlement batch is due and something is wrong.
What comes after this is card processor extraction (pulling WorldPay onto
the same Lambda and ECS pattern so the ACH and card rails share a unified
fault-tolerant layer) and eventually multi-processor routing that can fail
over between processors without the payment re-entering the submission queue.
That's a different series.
The payment infrastructure we built was not the most sophisticated payment
system anyone had built. It was an extraction from a Rails monolith under
production pressure, designed to be comprehensible to the engineers who would
debug it at 8am. The two panels that changed how we paged were not clever.
They were the minimum structure needed to stop asking the wrong question first.
Top comments (0)