We published a per-lookup overage rate on every paid plan from launch. None of it was ever charged to anyone.
The reporter existed. It ran nightly at 03:20, it selected the previous day's billable events, it sent them to the payment provider, and it exited zero. Every night. For weeks.
The 60-minute window
The provider rejects any usage event whose timestamp is more than an hour old:
events[0].timestamp: Timestamp cannot be older than 1 hour
A batch sent at 03:20 for the previous day had every single event refused. The job was not broken in any way a green checkmark could show you. It ran, it reported nothing, and it billed nothing.
The fix was the schedule, not the code: every 15 minutes instead of nightly.
The 200 that means nothing
The second one was worse. Reporting usage against a subscription whose product carries no meter returns 200. The events are accepted. The rows get marked reported. Nothing reaches the invoice.
That is indistinguishable from success at every layer you would normally check. We only found it by reading the meter's own count back, rather than trusting the response.
// reporting a 200 is not proof. the meter's count is.
const before = await meterTotal();
await report(events);
const after = await meterTotal();
The boundary nobody had tested
Overage is charged on the excess, not the allowance. At 20,001 lookups on a plan including 20,000, the customer owes for one lookup.
We had a function doing that arithmetic and no test executing it. The existing tests read the source as text and asserted on strings, which can prove a line exists but not that it returns the right number.
ok("20,001 charges for ONE lookup, not 20,001",
projectedSpend(starter, 20_001), 49 + 0.004);
We injected the bug before trusting the test — replaced the excess with the full count, and it failed with expected 49, got 129. An $80 error, stated as money.
What I would take from it
- A cron that exits zero is not a cron that worked
- Read the state back from the system you are integrating with, not the response it handed you
- Test the boundary where the arithmetic changes, and prove the test fails before you rely on it
None of these are clever. All three shipped anyway.
Top comments (1)
This is a great reminder that a successful API call is not the same as a successful business outcome.
One principle I’ve adopted for external integrations is to validate the downstream state whenever it’s practical. HTTP 200 only tells you the request was accepted—it doesn’t prove the intended business effect actually happened.
The billing bug and the missing meter both illustrate the same lesson: monitor business invariants, not just technical health. A green job can still represent a failed business process.