I wrote about the Electricity Planning Engine a little while back, including a timezone bug that made a correct price look "not found" after a database round trip. A few days later, Alex Shev left this comment:
Timezone bugs are brutal in planning engines because the result can look mathematically correct while being operationally wrong. Energy workflows especially need tests around boundaries, not just averages.
That is a genuinely sharp way to put it, and it is not just a comment about the bug I already wrote about. It is a comment about how I test the project in general, and I did not like how well it applied once I went and checked.
The part that stung a little
"Looks mathematically correct while being operationally wrong" is exactly what the original timezone bug was. PriceSeries::priceAt() threw a clean "price not found" error, which is arguably the good version of that failure mode: loud, easy to catch, hard to ship. A quieter version of the same class of mistake, off by one hour instead of missing entirely, would not throw anything. It would just return a plan that looks completely reasonable and is wrong the entire time it runs.
Alex's second point, boundaries over averages, is the one I actually had to go check rather than just agree with in the abstract. So I opened tests/Unit/Domain/Contract/PricingStrategyTest.php and looked at every hour used in every peak/off-peak assertion:
new DateTimeImmutable('2026-07-18 14:00:00') // peak
new DateTimeImmutable('2026-07-18 23:00:00') // off-peak
new DateTimeImmutable('2026-07-18 05:00:00') // off-peak
14:00, 23:00, 05:00. Every single one comfortably inside its window. None of them anywhere near the actual transition. The off-peak slot in the config is 22:00 to 06:00, and the comparison behind that lives in TimeSlot::contains():
// wraparound slot, e.g. 22:00 -> 06:00
return $minuteOfDay >= $this->startMinuteOfDay || $minuteOfDay < $this->endMinuteOfDay;
That >= versus < is exactly the kind of one-character decision that determines whether 22:00:00 itself is off-peak or not, and whether 06:00:00 itself is off-peak or already peak again. Nothing in the suite exercised either instant. A boundary mistake here would not crash, would not warn, would just silently bill one hour a day at the wrong rate, forever, until someone happened to notice their bill looked slightly off. That is Alex's point, precisely, and it was sitting in my own repo.
The fix
public function test_peak_off_peak_strategy_resolves_the_exact_boundary_minute_correctly(): void
{
$strategy = PricingStrategyFactory::fromConfig(ContractType::PeakOffPeak, [
'off_peak_slots' => [['start' => '22:00', 'end' => '06:00']],
'seasons' => [[
'label' => 'year_round',
'months' => range(1, 12),
'rates' => [
['slot' => 'peak', 'price_per_kwh' => 0.27],
['slot' => 'off_peak', 'price_per_kwh' => 0.20],
],
]],
]);
// Just before the off-peak window opens: still peak.
self::assertTrue($strategy->priceForHour(new DateTimeImmutable('2026-07-18 21:59:00'))->equals(Money::of(0.27)));
// The window opens exactly at 22:00:00: start is inclusive.
self::assertTrue($strategy->priceForHour(new DateTimeImmutable('2026-07-18 22:00:00'))->equals(Money::of(0.20)));
// Just before the off-peak window closes: still off-peak.
self::assertTrue($strategy->priceForHour(new DateTimeImmutable('2026-07-19 05:59:00'))->equals(Money::of(0.20)));
// The window closes exactly at 06:00:00: end is exclusive, already peak.
self::assertTrue($strategy->priceForHour(new DateTimeImmutable('2026-07-19 06:00:00'))->equals(Money::of(0.27)));
}
Four instants instead of three comfortable ones: one minute before the window opens, the exact opening second, one minute before it closes, the exact closing second. That is the whole idea of testing boundaries instead of averages, written down as assertions instead of just agreed with in a comment thread.
Proving the test actually tests something
A boundary test that would pass against a broken implementation is worse than no test, it is a false sense of safety. So before trusting this one, I broke the code on purpose: flipped the wraparound comparison from >= / < to > / <=, one character each, and reran just this test.
It failed immediately, on the 22:00:00 assertion, exactly where it should:
Failed asserting that false is true.
at tests/Unit/Domain/Contract/PricingStrategyTest.php:73
Then I reverted the one-character change and ran the full suite: 104 tests, 752 assertions, green. That failure-then-pass cycle is the only way I trust a new test is doing its job rather than just decorating the file with more green checkmarks.
Thanks, Alex
None of this was a bug in production, it was a gap in coverage that a bug could have hidden in later. Comments like Alex's are exactly how that gap gets found before the bug does instead of after. If you have opinions on where else this project's tests are testing averages instead of edges, the repo is open, and so are the issues.
Top comments (9)
"A boundary test that would pass against a broken implementation is worse than no test" — plus the failure-then-pass cycle being the only thing that earns your trust in it. I ran exactly that loop across a batch of gates this week, and it caught the thing you're describing in a spot I'd have sworn was fine.
Your one-character boundary is the part that lands hardest, because mine was literally that. I had an exclusion rule meant to keep a folder of deliberately-bad test fixtures out of a scoring pass. The pattern matched
tests/seedbut nottests/seed-clean— the trailing-cleanfell just outside the boundary the regex anchored on. One missing case, exact same shape as your>=vs<: six clean fixtures scored as real for weeks, mathematically running, operationally wrong, every run green. I only found it because I finally did your move — deliberately broke the gate and watched whether it went red. It didn't, for the boundary I'd never tested.The reader-found-the-gap half is the other thing I keep relearning: the boundary you didn't test is exactly the one you couldn't see, because if you could see it you'd have tested it. The cheapest audit I get is someone asking "what about the edge?" — and failure-then-pass is what turns their one-time question into a guardrail that stays.
Thanks a lot @fromzerotoship for sharing this story. That quote "A boundary test that would pass against a broken implementation is worse than no test" is spot on and deserves to be printed on a wall!
It’s wild how universal that single-character bug experience is. Your regex example (
tests/seedvstests/seed-clean) hits the exact same vein: false green tests giving a false sense of security while operating on wrong assumptions.The failure-then-pass cycle (red-green-refactor in its purest form) really is the only antidote to that confirmation bias. Glad the post prompted you to break that gate and catch the issue before production did! Thanks again for taking the time to write such a thoughtful response.
Thank you — but I have to correct the one generous part: I didn't catch it before production did. Production had been quietly living with it for weeks. Six clean fixtures were scored as real findings in every run during that window; the drill didn't prevent the bug, it performed the autopsy. That's the sharper version of your own line, I think: a gate that's silently broken doesn't just fail to catch things, it removes the possibility of catching them early, because "no failure reported" is exactly what it produces while broken.
On red-green-refactor — I think it's the same instinct with one difference in shape that took me a while to see. In TDD the red is a one-time event: you see it before the code exists, and after that the test carries itself. The red on a gate isn't an event, it's a perishable state. It proves the gate worked on the day you looked, and says nothing about whether the condition that makes it fire is still reachable next month. Mine drifted exactly that way — the check was falsifiable by design, but a widened pattern quietly made its failure mode unreachable, so "no failure observed" stayed true for the wrong reason.
Which points at the gap I still have: my seeded fixtures re-run automatically on every change, but the match under the gate is still something I do by hand, once, when I happen to think of it. So the very check I'm recommending is the one part of my setup that isn't protected against going stale. Next job is making the drill a thing that runs itself rather than a thing I remember. Thanks for the post that started it — it's been the most productive thread I've had this month.
Ah, the 'autopsy over prevention' realization that adds a whole layer of sharp truth to it. You nailed the precise danger: a silently broken gate doesn't just miss bugs; it manufactures unearned confidence. 'No failure reported' becomes the quietest kind of failure mode.
Your point on the perishable state of red vs TDD’s one-time event is a fantastic insight. A test that passed correctly on day 1 can drift into becoming completely unreachable on day 30, and standard CI will just smile and stay green the entire time.
Automating that 'drill': forcing the pipeline to periodically prove that the gate can still fail when it should, is a fascinating challenge. If you end up setting up a clean automated mechanism for self-testing those bounds (mutation tests in CI, canary fixtures, or synthetic failures), I’d love to read about it.
Thank you for sharing such deep, unfiltered context. This entire exchange has been one of the highlights of writing this series!
"Manufactures unearned confidence" is a better phrasing than mine — the gate isn't neutral while broken, it's actively issuing credit it hasn't verified.
On your request: I built it this week, so here's the shape while it's fresh. It's closest to mutation testing, but pointed one level up — mutation testing mutates production code to ask "does the test suite notice?", whereas this mutates the guards themselves to ask "does this check still report its own breakage?" Each run walks a list of cases: break one guard (hide a fixture, widen an exclusion pattern, age out an exception past its expiry), invoke the real check as a subprocess — not a copy of its logic — then restore. Eight cases, a few seconds.
Three implementation details that turned out to matter more than the idea:
And one trap for synthetic failures specifically, because it nearly bit me on a dead-man check: my first instinct was to simulate the failure for a future date, which would have written the "already alerted for that day" dedup record ahead of time — silently swallowing the real alarm if the checker actually died that day. The drill would have disarmed the thing it was drilling. Past dates only, and check which direction your synthetic failure writes into any suppression state before you run it.
Writing it up properly this week. This exchange is the reason it exists at all — you'll recognise most of it.
This is a brilliant extension of the concept.
"Manufactures unearned confidence" nails it completely a green gate that doesn't actually verify anything is a silent liability.
Your drill approach (testing the guards themselves rather than mutating production logic) is a fantastic layer of defense. Three points from your implementation really resonated:
Asserting the error message, not just the exit code: This is crucial. A non-zero exit from an unrelated side effect can easily disguise a broken assertion as a working one.
"Drilling the drill"/Meta-verification: Disabling a guard for real to verify the drill itself catches it is the ultimate application of the "must see it fail first" rule.
State/Deduplication traps with synthetic dates: That catch about future-dated alerts writing to suppression state and disarming real alarms in production is a golden operational insight.
I'm really looking forward to your write-up on this! Drop the link when it's live
Thanks — the write-up is queued, and I'll drop the link here the moment it's live.
One update worth adding to your third point, because it happened to me two days after that thread, and not in the code I'd spent a month hardening.
I wrote a throwaway script to analyze my own follower data. It fetched 473 profiles, hit an API rate limit partway through, silently discarded the 402 failed lookups with a
filter(Boolean), and printed confident percentages computed from the 71 that survived. No error. No exit code to inspect. A clean number that was, strictly speaking, about a population I never sampled.That's your first point wearing different clothes. I asserted the output and never asserted the denominator. So the script now refuses: if coverage drops below 90% it reports what it couldn't reach and exits non-zero instead of handing me a percentage.
The part that stings is that this happened in a domain with no drill at all, because I'd never filed "quick analysis script" under engineering. Every guard I own was in the other room.
It's a great reminder that the boundary between "throwaway script" and "production logic" is often just a illusion. The moment a script informs a decision, it's operating as production code and deserves the same assertion of inputs and coverage.
Can't wait to read the full article when it drops!
Your criterion is the part I was missing. I said the rigor doesn't travel between rooms; you gave the condition that says when it has to. "Does it inform a decision" is testable in a way "is this production" never was.
It caught me again about three hours after you wrote it, and the boundary turned out to be even thinner than "throwaway script."
I deployed some pages and ran a one-line shell check to confirm the hero images were live. It reported the image missing. I was moments from re-deploying to fix a deploy that was completely fine — the image was there; my regex didn't allow a hyphen in the build hash, and the filename had one. Not a script. A grep in a for-loop.
One refinement I'd offer, since this is now three for three: the dangerous ones are the checks whose negative result informs a decision. Mine said "missing," which told me to go do something, so I looked and found my own bug. If it had said "all fine," the decision it informed would have been to stop looking — and I'd never have learned it was broken. A check that can only tell you to act is much safer than one that can tell you not to.
Article goes up today. I'll drop the link.