Reconciliation broke twice a year for months before we found the pattern. Our settlement cutoff was defined as 23:00 local time, converted to UTC at deploy time and hardcoded. Worked fine for six months, then the clocks changed and the cutoff silently shifted by an hour relative to the acquirer's actual batch close.
The result: on the Sunday DST kicked in, roughly 40 minutes of transactions that should have landed in Monday's settlement file landed in Sunday's instead. Nothing errored. Both files were valid, both totals reconciled internally, but our ledger attributed a chunk of Monday's revenue to Sunday. Finance caught it three weeks later during month-end close, not us.
Fix was boring: store cutoff as a timezone-aware rule (America/New_York, 23:00, DST-adjusted), not a fixed UTC offset baked in at deploy. Also added a daily check comparing our computed batch boundary against the acquirer's actual file timestamp, alerting on drift over 5 minutes.
What surprised me is how long a boundary bug like this can survive. It doesn't fail loudly, it just quietly reassigns transactions across a date line twice a year, and every individual number still looks correct in isolation.
Anyone else hardcode a UTC offset somewhere they later regretted?
Top comments (0)