In July our finance team reconciled a week of refunds against the payment provider and found three hundred and twelve that had failed. Our refunds dashboard, which is built for exactly that, showed a hundred and nine for the same week. Every failure was in the logs.
The refund service counts failures with a Prometheus counter labelled with the provider and the reason. Like most code using a client library, it created each labelled series the first time it was needed: on the first failure with that reason, in that pod, the series appeared with the value 1. The dashboard and the alert both used increase over the last hour, summed across pods.
increase and rate work from differences between samples. A series that appears for the first time at 1 has no earlier sample, so the step from nothing to 1 is invisible to them. Prometheus cannot tell a counter that was just created at 1 from one that has always been 1. For a busy series that is nothing. Ours were not busy. Failures were spread over forty pods and fourteen reasons, and the pods were replaced on every deploy, five or six times a day. Most combinations of pod and reason saw one failure in their whole life, sometimes two. So most failures were the first of their series, and the first was the one we lost.
The alert was worse than the dashboard. It fired at more than five failures in an hour, and in the hour the provider's refund endpoint misbehaved it saw about a third of them.
The fix is the one the Prometheus documentation recommends. At startup the service creates every combination of provider and reason it knows about, with the value zero, so the first real failure is a step from 0 to 1 that increase can see. Reasons we do not know about go into a single other label, which is also created at zero. A unit test fails if any counter in the service is created lazily. And the refunds dashboard now has a second panel fed by the nightly reconciliation, so a metric and a count from the ledger sit side by side and a gap between them is visible.
A counter tells you how much it went up. It cannot tell you about the moment it came into existence, and for rare events that moment is most of the story.
– Sergey Shinder
Top comments (0)