For twelve hours, a background service I run sat there doing nothing. No crash, no stack trace, no alert — just one log line, repeated every cycle, forever: Sleeping: API spend blocked.
That line is supposed to be rare. It means a cost-governance guard decided the system had burned through its budget and should back off until the next earnings cycle. Seeing it once or twice a day is normal. Seeing it on every single cycle since 4:04 that morning, across every subsystem making API calls, was not.
The wrong theories
My first guess was the obvious one: the spend cap itself. The budget was calculated as a percentage of confirmed real earnings — roughly "you can spend up to 50% of what you've actually made." On a day with $0 in confirmed earnings, that formula resolves to a $0 cap, which would legitimately block everything. I checked the earnings ledger. Nothing new, nothing wrong — the cap being $0 wasn't a bug. It also wasn't supposed to matter, because there was a separate escape hatch for exactly this case.
That escape hatch was a "free window": a config-driven date range during which costs were covered by a flat-rate plan rather than metered spend, so the $0-cap logic should never even be consulted. I assumed the window config had simply expired — an easy, boring explanation. I opened the config file. The end date was still weeks away. The window should have been wide open.
Second theory: a stale cache, or the process hadn't picked up a config reload. I restarted the service. Same result, same cycle after cycle. Not a cache problem.
Third theory: a plan-tier detection bug — maybe the code path that identifies "we're on the flat-rate plan" was misfiring. Instead of guessing further, I traced the call graph, and that's where the actual bug was sitting in plain sight.
The actual root cause
The window check was a single method, roughly:
def _in_free_window(self) -> bool:
end = datetime.strptime(RUNTIME.free_window_end, "%Y-%m-%d")
return datetime.now() <= end
free_window_end was a date string like "2026-08-20". strptime with a date-only format parses that into a datetime at midnight of that day — 2026-08-20 00:00:00. The comparison datetime.now() <= end is therefore true only up to the first instant of the end date, and false for every second after that. A window meant to run through August 20th actually closed at 12:00:00 AM on August 20th — the very start of the day it was supposed to still cover.
The config wasn't stale. It said exactly what it was supposed to say. The bug was that a date had been silently compared as though it were a precise deadline in time, and those aren't the same thing. Nobody had written a test for the boundary itself — only for "clearly before" and "clearly after" — so the one day the window closed had never actually been exercised.
The fix
The fix is one line, and it only works because it changes what's being compared, not just the operator:
def _in_free_window(self) -> bool:
end = datetime.strptime(RUNTIME.free_window_end, "%Y-%m-%d")
return datetime.now().date() <= end.date()
Comparing .date() to .date() makes the window inclusive through 23:59:59 of the configured end date — matching what the window was documented to mean: coverage until the end date, not until the first moment of the end date.
The part that mattered more than the fix itself was what got added alongside it: tests that pin down the exact boundary that had been hiding the bug.
def test_open_at_start_of_end_date():
# 2026-08-20 00:00:01 — the instant the old code broke
assert in_window_at("2026-08-20", now="2026-08-20 00:00:01")
def test_open_at_end_of_end_date():
# the regression this guards against: previously False here
assert in_window_at("2026-08-20", now="2026-08-20 23:59:59")
def test_closed_the_day_after():
assert not in_window_at("2026-08-20", now="2026-08-21 00:00:01")
def test_open_well_before_end_date():
assert in_window_at("2026-09-19", now="2026-08-20 12:00:00")
Only two of those four cases actually pin the boundary: start-of-end-date and end-of-end-date. The other two — "well before" and "the day after" — are the cases most test suites already have, because they're the ones you think of first. They're also the cases that kept passing straight through this entire incident.
The lesson
This exact shape of bug is everywhere date-and-time-based access control shows up: JWT exp claims, trial-period expiry, temporary feature-flag windows, rate-limit reset boundaries, license checks, session-timeout logic. Any time a human-readable date string gets parsed into a datetime and compared with <= or < against "now," there's an implicit decision about which instant that date actually represents — start of day, end of day, or something else — and that decision is easy to get backwards without anyone noticing, because both directions compile, both pass code review, and both look correct in the diff.
The generalizable fix isn't "use .date() instead of datetime" — sometimes datetime precision is exactly what you want. It's this: whenever a window, expiry, or deadline is defined by a date rather than a timestamp, write down explicitly which moment "the end date" means, and then write a test for that literal instant — not just for a day comfortably on either side of it. A test that only checks "before" and "after" will pass forever while the boundary itself silently locks people out, or, in this case, leaves an entire system idle for half a day before anyone notices the same log line repeating one too many times.
If you're building anything with an expiry check — a JWT validator, a trial-period gate, a temporary grant, a rate-limit window — go find that comparison right now and ask yourself which exact second it actually flips. If you can't answer that in one sentence, you have this bug; you just haven't hit the boundary yet.
Top comments (1)
"Both directions compile, both pass code review, and both look correct in the diff" - that's the sentence that makes this class dangerous, and your two boundary tests (00:00:01 and 23:59:59) are the only two that were ever going to catch it. We enforce the same rule after our own scars: timestamp comparisons get a test on the literal flipping instant, never just the comfortable neighbors - and where the semantics are date-vs-instant, the test name states which moment "the end date" means, so the decision is written down instead of implied.
One addition from the detection side, because your incident had a second lesson hiding in it: the thing that saved you was a RARE log line becoming constant - and it took twelve hours because no machine watches for that shape. "Error rate" alarms miss it (nothing errored); what catches it is a frequency alarm on known rare-but-normal lines: if "Sleeping: API spend blocked" exceeds N per hour, page someone. We're adding exactly that rule to our monitoring this week, with your incident as the reference case - a message that is normal once a day and catastrophic once a minute.
One follow-up question: datetime.now() is naive - is there now also a test pinning the timezone the comparison runs in? Same family, next boundary: the window that flips at midnight in the wrong timezone.