Our payment reconciliation consumer stopped at 04:12 and did not start again. By the time anyone was awake there were nineteen thousand unprocessed settlement records and a finance team that could not close the previous day.
The provider had introduced a new settlement status, PENDING_REVIEW, for transactions their fraud system wanted a second look at. They had announced it in a changelog three weeks earlier. Our deserialiser mapped that field to an enum and was configured to fail on unknown values, which is a setting I have defended in code review more than once. Strict parsing catches typos and schema drift early. It also, here, threw on the first record that carried the new value.
That would have been survivable. The part that turned it into an outage is that we read those records in pages of five hundred and deserialised the page as a whole. One unknown enum in one record failed the entire page, the page was not acknowledged, the consumer retried it, failed identically, and after the retry budget the whole consumer shut down rather than skipping forward. Sixteen affected records took down nineteen thousand good ones.
We changed three things, in order of how much they mattered. Records are now deserialised individually, and a record that cannot be parsed goes to a quarantine topic with the raw payload attached while the rest of the page proceeds. That alone would have turned this into a report instead of an incident. The enum now has an explicit UNKNOWN fallback, and any record that lands on it is quarantined and alerted on rather than silently accepted, because I still do not want unknown values flowing through the reconciliation logic as though we understood them.
Third, and this is the boring one that will actually prevent the next occurrence, we have a scheduled job that fetches the provider's published schema and diffs it against the one we build against, and their changelog feed now posts into our channel. Somebody reads it now because it appears where we already are.
A closed set of values is only closed on your side of the integration. On their side it is a list somebody can add to on a Tuesday.
– Sergey Shinder
Top comments (0)