A Pairing Hour on a Timezone-Flaky Test: Three Dead Ends and the One Seam We Kept

The failing test only broke between 23:00 and 23:05 UTC. Locally it passed every time, on every laptop, for two weeks. Then a senior engineer sat down for a one-hour pairing session and spent the first ten minutes asking questions instead of opening an editor.

What follows is that hour, kept close to how it happened: the questions, the three fixes we abandoned, and the single change that survived review. The artifact is a probe script and a clock seam small enough to paste into a real repository.

The first ten minutes were questions, not code

The senior did not read the assertion first. They asked, in this order:

1. What does the test actually assert — a wall-clock time, a duration, or an ordering?
2. Which clock produced the value under test: the system clock, a library clock, or a clock inside the framework?
3. Whose offset matters — the machine's TZ, the database session's time zone, or the scheduler's own?
4. Who else in the codebase already reads time, and can they be changed without touching behavior?

The answers were uncomfortable. The test asserted a formatted string. The code read new Date() directly in three modules. Nobody could say which TZ CI used, because the workflow set it in one job and inherited it in another.

That last answer explained the 23:00 window without explaining the failure. A vague explanation is not a diagnosis.

Reproduce before you read the stack trace

We wrote a shell probe that runs the same spec under several wall-clock offsets, repeatedly, and prints a matrix instead of a wall of logs. It is deliberately dependency-light: one script, one runner variable, no framework hooks.

#!/usr/bin/env bash
# probe_tz.sh — run one spec under several wall-clock offsets.
set -euo pipefail

SPEC="${1:-tests/scheduler.spec.ts}"
RUNNER="${RUNNER:-npx vitest run}"
RUNS="${RUNS:-3}"
ZONES=(UTC Asia/Tokyo America/Sao_Paulo Pacific/Chatham)

fail=0
for tz in "${ZONES[@]}"; do
  for i in $(seq 1 "$RUNS"); do
    printf '%-20s run %d: ' "$tz" "$i"
    if TZ="$tz" $RUNNER "$SPEC" >/tmp/probe-$tz-$i.log 2>&1; then
      echo PASS
    else
      echo FAIL
      fail=1
    fi
  done
done

exit "$fail"

Two details matter more than the code. First, Pacific/Chatham is in the list on purpose: a 45-minute offset plus DST catches arithmetic that assumes whole-hour offsets. Second, the script keeps per-run logs under /tmp, so a failure is inspectable without re-running the matrix.

If RUNS is left at 3, a flaky result that appears once out of twelve looks like noise. Set RUNS=5 or higher before trusting a green matrix — a single pass proves nothing about a race.

Dead end 1: add retries to the test

The first suggestion was the cheapest one: wrap the assertion in a retry helper. It turned red into green in four minutes.

It also deleted the signal. The flake was deterministic given an offset, and retries converted a reproducible boundary bug into a probabilistic one that would eventually surface in production at the same hour, with real schedules attached. The senior's note was blunt: retries are for infrastructure you do not control, not for your own arithmetic.

We reverted it in the same session. It never reached review.

Dead end 2: a global fake timer

The second attempt replaced the whole timer system in the test process, froze time at a fixed instant, and advanced it manually. The spec passed. So did three unrelated specs that had been quietly relying on real timers, because the fake timer also stopped a retry backoff and made two slow tests look fast.

Worse, it hid the original bug. Freezing the clock removes the offset, and the offset was the bug.

A related trap: assigning process.env.TZ inside a test after the process has started. In current Node.js runtimes the time zone is resolved early, so a mid-test assignment is unreliable and can pass on one machine while failing on another. A probe matrix set from the shell avoids that entirely.

Dead end 3: a production freeze flag

The third proposal was the most tempting for a deadline: read an environment variable in the scheduler and let tests pin the current time.

We rejected it for two reasons that are cheap to state and expensive to discover late. A production code path now branches on test configuration, which means the code under test is not the code that runs. And anyone who sets that variable in the wrong environment silently stops scheduled work — a flag that looks like a convenience and behaves like an outage switch.

The decision we kept: one clock seam

The change that survived is small, explicit, and boring. Time enters the scheduler through a parameter with a default, and nothing else in the module calls new Date() at the top level.

// src/clock.ts
export interface Clock {
  now(): Date;
}

export const systemClock: Clock = { now: () => new Date() };

// src/scheduler.ts
import { type Clock, systemClock } from './clock.js';

export function nextRun(clock: Clock = systemClock): Date {
  const now = clock.now();
  const next = new Date(now.getTime());
  next.setUTCHours(2, 0, 0, 0);
  if (next.getTime() <= now.getTime()) {
    next.setUTCDate(next.getUTCDate() + 1);
  }
  return next;
}

// tests/scheduler.spec.ts
import { describe, expect, it } from 'vitest';
import { nextRun } from '../src/scheduler.js';

const at = (iso: string) => ({ now: () => new Date(iso) });

describe('nextRun', () => {
  it('returns the next 02:00 UTC when now is before it', () => {
    expect(nextRun(at('2026-03-08T01:30:00Z')).toISOString())
      .toBe('2026-03-08T02:00:00.000Z');
  });

  it('rolls to tomorrow when now is exactly 02:00 UTC', () => {
    expect(nextRun(at('2026-03-08T02:00:00Z')).toISOString())
      .toBe('2026-03-09T02:00:00.000Z');
  });
});

The two assertions above are boundary cases, not coverage. The matrix script is what proves offset independence, and it runs against the same suite in CI as a single job with RUNNER pinned to the project's test command.

What the seam costs

• One interface file, one default parameter, and a reviewer who can see where time enters the module.
• A small refactor if other modules read time directly. Do that separately, in its own commit, so the behavioral change stays reviewable.
• No new dependency, no test-only branch in production code, and no configuration flag that can be mis-set.

Where a free model and a disposable server fit the hour

The senior ran the pairing hour; the tooling was mine. I used MonkeyCode's free model access to draft candidate hypotheses for the flake — five plausible causes, written down before anyone touched the code — and I used the free server option as a throwaway Linux box to run the TZ matrix somewhere other than my laptop, because a local machine with a cached build is not a clean environment.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the model was good for: enumerating causes quickly, including offset-aware ones I had not considered, and drafting the first version of the probe script. What it was not good for: deciding which hypothesis was true. Two of its five suggestions were dead ends 1 and 3 above. The model generated options; the shell matrix and the senior's review decided between them.

Honest limits of that setup, as I experienced it:

• The free server option is a shared, ephemeral environment. Treat it as a disposable probe box, not a place for endurance runs or timing-sensitive benchmarks.
• Do not paste credentials, customer data, or private repository contents into any model prompt. The probe matrix needs an environment variable and a test command, nothing more.
• Free-tier terms, allowances, and available models change. Check the project's own current documentation rather than a screenshot or a blog post, including this one.

A decision table for the next flake

Symptom	Adopt	Park	Revert
Fails only in a time window	Probe matrix + clock seam	—	Retry wrappers
Fails only on one machine	Pin the offset in CI	Containerize the runner	Global fake timers
Passes with fake timers, fails without	Inject the clock at one seam	Broaden the matrix	Freeze flag in production code
Fails under load, not under time	Reproduce with a concurrency probe	Add a soak job	Anything clock-related

Limits, and who should skip this

This approach fits deterministic boundary bugs that a wall-clock offset can expose. It does not fit every flake. If the failure comes from real concurrency, network timeouts, or a shared fixture, a TZ matrix will stay green and teach nothing.

Skip it if your tests run against regulated data that cannot leave a managed environment, if you need dedicated hardware for reproducible timing, or if the team cannot commit to keeping one CI job for the matrix. A probe script that nobody runs is worse than no script, because it suggests coverage that does not exist.

What went into the pairing notes

One line, written by the senior before the hour ended: the bug was in the arithmetic, the fix was in the seam, and the retries were the expensive part.

If your next pairing hour loses its first thirty minutes to environment setup rather than to the bug, that is the workflow worth testing before you spend anything on tooling.