Series — Building ACH on AWS: 0: System Design · 1: Sidekiq Latency · 2: Extraction Boundary · 3: Shadow Traffic · 4: What Broke · 5: Lambda/ECS Split · 6: SQS FIFO · 7: Instrumentation · 8: Triage Panels
During an ACH incident, the first question on-call asks is: is this us or ProfitStars? Answering it in under a minute (at 2am, with payments backing up) requires the right instrumentation to already exist. Work backwards from that question and you get a clear picture of what to build. Work forwards from "let's add metrics" and you end up with a noisy dashboard that doesn't answer anything under pressure.
This post covers both: the metrics we added to the payment state machine, and the ones we deliberately did not. The skip decisions are just as consequential as the add decisions.
What we instrumented, and where
The state machine had four states (pending, submitted, confirmed, failed) and handled both ACH and card payments. The instrumentation rule we landed on early was: put the telemetry at the transition layer, not scattered across individual controllers. Every state change in the system went through the same code path. Instrumenting there meant one place to maintain and no gaps from a controller that someone forgot to update.
From that single layer, we added three categories of metrics in Datadog.
Transition counters. A counter for every state change, tagged by destination state and payment method (ACH vs card). This gave us a real-time feed of what the system was doing. A sudden drop in pending -> submitted transitions meant new ACH submissions had stopped. A spike in submitted -> failed transitions meant something was wrong at the processor boundary. Both showed up immediately without waiting for a dashboard refresh.
Time-in-state histogram for submitted. This one was specific to ACH. Once a payment entered "submitted," it waited for ProfitStars to send an async settlement callback: a T+1 window that could span a full business day. We tracked how long each payment spent in that state as a histogram. The P50 was our baseline: under normal conditions, most payments settled in about 24 hours. That number barely moved week to week.
The P99 was the signal. If the P50 was normal but the P99 started climbing, we had a tail of individual payments stuck in submitted: the kind of thing that happens when a callback was dropped, a message sat in the retry queue too long, or a specific firm's payments hit an edge case. Different root cause from a general slowdown. If both P50 and P99 climbed together, ProfitStars was running slow across the board. Watching both in tandem let us tell the difference in one glance.
Deduplication counter. The SQS FIFO queue used processor_event_id as the deduplication key, so duplicate callbacks from ProfitStars were automatically rejected. We added a counter on those hits anyway, not to prevent duplicates (FIFO did that) but to know how often ProfitStars was sending them. A spike in deduplication hits was useful context when we were already looking at an incident. Low and steady meant nothing interesting was happening.
Pending-aging alert on the allocator. The allocator ran after a payment confirmed, applying the payment amount against invoices and writing ledger entries inside a single database transaction. But the payment record's state had already transitioned to "confirmed" before the allocator ran. If the allocator crashed mid-write, you had a confirmed payment with pending ledger entries: the invoice still showed as outstanding, the client had been charged, and the firm's bookkeeper had no idea why. The alert fired if any confirmed payment had pending ledger entries more than 5 minutes after confirmation. Normal allocation ran in milliseconds. Five minutes meant the allocator had gotten stuck, which was worth a page.
All four of these lived at the state machine transition layer. One place to look during an incident.
SQS queue health. Three alerts on the settlement queue, each answering a different question. ApproximateNumberOfMessagesVisible above a threshold for more than 5 minutes: the worker fleet wasn't keeping up: capacity question, usually an ECS scaling lag. ApproximateAgeOfOldestMessage above 30 minutes: one message was stuck across multiple retry cycles: single-payment investigation. DLQ depth above 0: a payment had exhausted all retries: high severity, requires someone to pull the message and make a judgment call about whether to replay or escalate.
What we did not add
Three categories of metrics we scoped out during design, each for a different reason.
Per-firm payment volume. MyCase served hundreds of law firms. Tagging a transition counter with firm_id would have produced a separate metric series per firm per state: cardinality that would have blown up Datadog costs fast. The operational value was also low: if a specific firm was having a problem, we could find it from the DLQ message, which contained the payment_id. The metric didn't need to carry the firm label.
Payment amounts in metric labels. PCI-adjacent risk. Even a histogram of amounts as label values starts to look like financial data in your metrics infrastructure. We tracked counts and latencies. If we needed to know the dollar value of stuck payments, we queried the database.
Individual webhook payload contents. A ProfitStars callback carries enough information that logging its fields in Datadog carelessly would create a security surface area with no operational benefit. The behavior was what we needed to observe: did the callback arrive, was it a duplicate, did it advance the state machine. The payload content was irrelevant to any question we were trying to answer in an incident.
The rule we landed on: metric labels should describe behavior, not payload. State, payment method, transition direction, queue depth: behavioral. Firm ID, amount, response body: payload. The line was clear once we named it.
Three noise sources we fixed
Roughly 35% of our alerts were noise before a focused cleanup. Three sources.
ECS auto-scaling task replacement. When the ECS worker fleet scaled out, new tasks spun up but weren't immediately available; they had to pass health checks first. During that window, our "payments unavailable" alert was checking the task count, seeing the gap, and firing. By the time on-call looked at it, the new tasks had passed their checks and the system was fine. Fix: we added a 3-minute evaluation window before the alert fired. A genuine outage sustained for 3 minutes would still page; a healthy scale event resolved before 3 minutes elapsed silently.
P50 alerting on normal ProfitStars variance. ProfitStars, like any external HTTP dependency, had some natural response-time variance. We had set our timeout alert threshold at approximately the P50 response time. The result was that any time ProfitStars ran slightly slower than its median, which happened regularly, we paged. We were alerting on normal. Fix: we moved the threshold to P99 plus a 20% buffer. Real degradation showed up there. Routine variance didn't.
T+1 settlement batch anomaly detection. Every morning around 8am, the overnight ACH batch arrived from ProfitStars. All those callbacks came in at once, SQS depth spiked, worker CPU climbed: completely predictable, every business day. Datadog's anomaly detection treated it as an incident every time. Fix: a scheduled maintenance window plus a separate, higher threshold during that window. The spike remained visible in the dashboard. It stopped paging anyone.
After fixing all three, alert volume dropped 35%. More importantly, the alerts that did fire were nearly always real.
What this enables
With those metrics in place, the "us or ProfitStars" question had a two-panel answer. External HTTP client error rate up, internal Rails exception rate flat: ProfitStars problem, check their status page. Internal exception rate up, ProfitStars healthy: our problem, check recent deploys and the DLQ. Two panels, two completely different runbooks, no ambiguity.
Part 8 is about how those panels were built and what one specific incident looked like when the split let on-call route correctly in two minutes instead of twenty.
Top comments (0)