The test that failed every morning and passed every afternoon

Originally published on the WatchNext blog.

While making an unrelated change — adding a processor to a privacy page and a link to a footer — the test suite came back with eleven passes and one failure:

- 1
+ 2

 ❯ tests/airDate.test.ts:125:47
    125|     expect(getAirDateDaysDiff(plus(1), "US")).toBe(1);
       |                                               ^

An episode airing tomorrow was being counted as two days away. That is about as load-bearing as a bug gets in an app whose entire purpose is telling you when the next episode airs.

It turned out the countdown was fine. The test was wrong, in a way that made it fail for roughly a third of every day and pass for the rest.

First: prove it isn't your change

The edited files were a privacy page, a footer and a markdown document. The failing test covers air-date arithmetic. Those are obviously unrelated — but "obviously unrelated" is a hypothesis, and the cost of checking it is one command:

git stash && npx vitest run   # same failure
git stash pop

Identical failure with the changes removed. That single result reframes everything that follows: this is not a regression being debugged, it is an existing defect being discovered. Without it, the natural next move is to start reading your own diff for a cause that was never in it — and the worst outcome of that search is "fixing" code of your own that was correct.

Then: rule out the machine

A test about day counting failing intermittently points at timezones, so the first suspect is the machine's own zone. That is easy to test by simply telling the process it lives somewhere else:

TZ=Europe/Dublin     npx vitest run   # 1 failed
TZ=America/New_York  npx vitest run   # 1 failed

Both fail, identically. That is a useful negative: the host timezone changes what the process calls local time, but it does not change the instant at which the test runs. So the hidden dependency is not where the test runs. It is when.

Two calendars, one comparison

Here is the test as written. It builds "tomorrow" and "yesterday" by adding and subtracting a day in milliseconds, then trimming the result to a date:

const today = new Date();
const iso = (d: Date) => d.toISOString().slice(0, 10);
const plus = (n: number) => iso(new Date(today.getTime() + n * 86400000));

expect(getAirDateDaysDiff(plus(1), "US")).toBe(1);
expect(getAirDateDaysDiff(plus(-1), "US")).toBe(-1);

That looks unimpeachable, and it contains the whole bug. toISOString() always formats in UTC. So the date handed to the function is "tomorrow according to UTC".

The function, though, deliberately does not count days in UTC. It counts them in the show's own country's timezone, and for a US show that means America/Los_Angeles. That choice is not an accident or an oversight — it is the fix for an earlier bug where a show's card and its notification disagreed about whether the same episode aired today or tomorrow, because one of them mixed the viewer's timezone into the comparison.

So the test measures from one calendar and the function measures from another. For most of the day they agree. For the hours when UTC has already rolled over to a new date and Los Angeles has not, they do not:

Moment	Test asks about	Function's today	Result
15:00 UTC, 11 Sept	12 Sept	11 Sept (LA)	1 ✓
06:39 UTC, 12 Sept	13 Sept	11 Sept (LA)	2 ✗

At 06:39 UTC on 12 September it is still 23:39 on 11 September in Los Angeles. UTC's "tomorrow" is 13 September. Los Angeles's "today" is the 11th. Two days apart, and the assertion wanted one.

Which means it fails on a timetable

The failure window is exactly the offset between the two zones: seven hours while Los Angeles is on daylight time, eight hours when it isn't. Midnight UTC to around 07:00 UTC, every single day. After that it goes green again on its own.

Two things about that are worse than an ordinary broken test. The first is that it presents as a feature bug — the assertion says an episode airing tomorrow is being counted as two days out, which is precisely what a real off-by-one in the countdown would look like. The second is that re-running it later makes it pass, which quietly teaches everyone that it is "flaky" and can be re-run rather than read.

The fix, and what it deliberately doesn't cover

The repair is to build the fixture in the same frame the function measures in, rather than in UTC:

const zone = timeZoneData.find((e) => e.iso_3166_1 === "US")?.ianaTimeZone;
const plus = (n: number) =>
  DateTime.now().setZone(zone).plus({ days: n }).toISODate() as string;

The zone is looked up from the same table the application uses, so the test does not hardcode a mapping that could later drift out of step with the code.

That has a real cost worth naming: because the fixture now derives its zone from the same table as the code under test, this test can no longer catch a wrong entry in that table. It is not trying to. Its job is the day arithmetic, and covering the mapping is a different test with fixed, hand-written dates. A test that quietly tests two things is how you end up with one that fails for reasons its name doesn't explain.

Checking the fix didn't just silence it

The easiest way to make a failing test pass is to adjust it until it agrees with whatever the code currently does. That is not a fix; it is a deletion with extra steps. So the fixed test was checked the same way the original air-date work was — by breaking the code on purpose and confirming the test notices.

The mutation was to reintroduce exactly the bug this block of tests is named after, replacing the zone-aware "now" with a plain local one:

- const now = zone ? DateTime.now().setZone(zone) : DateTime.now();
+ const now = DateTime.now();

Red again, as it should be. The test still guards the thing it was written to guard; it simply no longer reports a failure that depends on what time you asked.

The general version

Any test that calls new Date() has an input that isn't written down anywhere in it. Most of the time that input is harmless. It stops being harmless the moment the code under test cares about which calendar it is using — and if you are dealing with schedules, billing periods, delivery estimates or anything else with a day boundary in it, your code cares.

There are two honest ways out. Freeze the clock, so "now" is a fixed value you chose. Or, as here, build your fixtures in the same timezone the code measures in, so there is only one calendar in the comparison.

What does not work is the thing that feels most neutral: reaching for UTC in a test because UTC feels like the absence of a timezone. It isn't. In a codebase that reasons in a show's local calendar, UTC is simply a third timezone that nobody asked for — and it will agree with the right answer often enough, and for long enough, to get committed.

After this went up: a reader pushed on the caveat

The note above admits that the repaired test can no longer catch a wrong entry in the timezone table, because it now reads that table itself. A reader asked the obvious follow-up: is anything pinned outside that frame, so the two calendars can still disagree on purpose?

They also supplied a better description of the problem than the one in this post's title — a test with a schedule is not flaky, it is deterministic and on a timer — along with the observation that a retry policy hides exactly that, forever.

The honest answer was "partly, and by accident", which is worth showing as a measurement rather than an opinion. Pointing US at Europe/Berlin fails three other tests, because the day-shift logic compares the origin's offset against Dublin and Berlin is not behind it. But pointing it at America/New_York — still behind Dublin, still entirely plausible — failed nothing at all. The mapping was pinned only insofar as a zone sits east or west of Ireland.

Three things are pinned now.

A frozen instant, which was the reader's suggestion, with one correction to it. The obvious choice of a small-hours UTC time does not discriminate: at 02:00 or 03:00 UTC, Los Angeles and New York are still on the same calendar day and agree on every answer here. 06:00 UTC is where they split — 23:00 on the 11th in Los Angeles, 02:00 on the 12th in New York.

vi.setSystemTime(new Date("2026-09-12T06:00:00Z"));

// In Los Angeles it is still the 11th, so the 12th is tomorrow.
expect(getAirDateDaysDiff("2026-09-12", "US")).toBe(1);
expect(getAirDateDaysDiff("2026-09-13", "US")).toBe(2);

Both halves hardcoded, nothing read from the table. The America/New_York drift now fails this test, and only this test.

The daylight-saving boundary — where checking the claim turned up that we had been wrong about our own code. The assumption was that Math.round was what kept day counting correct across a 23-hour day. It isn't: Luxon's difference between two startOf("day") values is calendar arithmetic, so the transition was already handled and the rounding was doing nothing. Worth pinning anyway, because the obvious simplification does break — a plain millisecond difference across Los Angeles's spring-forward is 0.9583, which floors to zero. An episode airing tomorrow, reported as airing today, one day a year, in one timezone.

A boundary case better than a frozen instant. Côte d'Ivoire sits at UTC+0 all year and observes no daylight saving. Ireland does. So the same origin country is behind the release zone during Irish summer time and exactly level with it in winter — and the shift correctly applies in one case and not the other. That pair cannot be satisfied by a hardcoded list of "western" origin countries, whichever way the list is written, and it puts the Dublin transition itself under test, since that is the thing that moves the answer.

The last part of the question was whether CI runs the suite inside the failure window. It wasn't running the suite at all — there was no test job, only a nightly data-refresh workflow, which is a large part of how a test that failed for seven hours a day survived as long as it did. There is one now, and it deliberately does not pin a timezone: these tests are supposed to hold wherever and whenever they run, and forcing a zone would hide the exact class of bug they exist to catch.

---

WatchNext is a deliberately simple TV tracker: it tells you when the next episode of your favourite shows airs, for any show, in any country — and nothing else.

Comments

[Raknaos]

The part I keep pointing people at in this write-up is that re-running it later makes it pass, which teaches the whole team to re-run instead of read. A test with a schedule is not flaky, it is deterministic and on a timer, and the seven-hour window you derived is exactly the kind of thing a retry policy hides forever.

One tension in the fix: the fixture now reads the zone from the same table the app uses, so if that mapping drifts the test agrees with the wrong behaviour instead of catching it. Do you keep anything pinned outside that frame - a frozen instant at 02:00 UTC on a DST boundary, or a CI job that runs the suite inside the failure window - so the two calendars can still disagree on purpose?

[Eugen Taranowski]

"A test with a schedule is not flaky, it is deterministic and on a timer" — I'm stealing that. It's the better framing, because "flaky" is exactly the word that licenses the retry.

You found a real gap, so I measured it rather than guessing. An obviously wrong mapping does get caught: US → Europe/Berlin fails three existing tests, since the shift compares the origin's offset against Dublin. But US → America/New_York — still behind Dublin, still plausible — failed nothing. Twelve green. The mapping was only pinned insofar as a zone sits east or west of Ireland, and the day-counting test couldn't see it by construction, exactly as you said.

Three things now pinned outside that frame.

A frozen instant, with one wrinkle on your suggestion. 02:00 UTC doesn't discriminate — LA and New York are on the same calendar day there and agree on everything. 06:00 UTC is where they split (23:00 on the 11th in LA, 02:00 on the 12th in New York):

vi.setSystemTime(new Date("2026-09-12T06:00:00Z"));
expect(getAirDateDaysDiff("2026-09-12", "US")).toBe(1);
expect(getAirDateDaysDiff("2026-09-13", "US")).toBe(2);

Both halves hardcoded, nothing read from the table. The New_York drift now fails this and only this.

The DST boundary — and here I was wrong about my own code. I assumed Math.round was load-bearing across transitions. It isn't: Luxon's diff between two startOf("day") values is calendar arithmetic, so a 23-hour day already counted as one day. Worth pinning anyway, because the obvious simplification does break — a millisecond diff across Los Angeles's spring-forward is 0.9583, which floors to 0. An episode airing tomorrow reported as airing today, one day a year, in one timezone. Mutation-checked: swapping in (b - a) / 86400000 fails spring-forward and nothing else.

A better boundary case than a frozen instant, as it turns out. Côte d'Ivoire is UTC+0 all year with no DST; Ireland has it. So the same origin country is behind the release zone in Irish summer and level with it in winter — and the shift correctly applies in one and not the other. No hardcoded list of western origin countries can satisfy both, and it puts the Dublin transition under test, since that's what moves the answer.

On CI: there wasn't a test job at all, only a nightly data-refresh workflow — which is how a test failing seven hours a day survived as long as it did. There is one now, and it deliberately does not set TZ: the pinned instants are supposed to hold wherever and whenever they run, and forcing a zone would mask the exact class of bug they exist to catch. Runners are UTC, which is the window the original flake lived in anyway.

I'd still argue the frozen instant beats a job scheduled inside the failure window, for your reason rather than mine: a window job would have caught this, but it would also have been a second thing that only fails at certain hours.