Defusing Time Bomb Tests using Randomization

Flaky tests are generally something to avoid, but even worse are tests that start failing a year after they merge. These tests are time bombs waiting to go off when we're least prepared (e.g. in the new year). What if adding chaos into our test environments actually helped surface these issues earlier in the development cycle?

Note: All examples are written for Jest tests, but the concepts can be adapted for other testing frameworks/languages.

Time in Tests

You may have already written a test that worked when you wrote it and even worked in CI, but it stopped working at a later date - especially around New Year's when offices are closed.

Here's an example:

// relies-on-time.js
export function isInPast(date) {
  return new Date() >= date;
}

// relies-on-time.test.js
describe("relies-on-time#isInPast", () => {
  it("should return false for dates in the future", () => {
    expect(isInPast(new Date("2020-12-31T00:00:00.000Z"))).toBe(false);
  });
});

At time of writing, this test passes, but on Jan 1, 2021, the test will begin failing. This isn't ideal because this code could have merged long ago. By the time it begins failing, it may block application deployments and we've lost some context on the intended behavior.

Using Fake Timers

One solution is to use fake timers to mock the date. If you've spent a lot of time writing Jest tests, you've even likely mocked Date.now to set the current date.

As of Jest 26.1.0 it includes the jest.setSystemTime(now?: number | Date) function to mock the system time as long as you're using the "modern" setting.

Using fake timers and jest.setSystemTime, we have a test that will now pass regardless of the current date:

// relies-on-time.test.js
describe("relies-on-time#isInPast", () => {
  it("should return false for dates in the future", () => {
    jest.useFakeTimers("modern");
    jest.setSystemTime(new Date("2020-11-20T00:00:00.000Z"));

    expect(isInPast(new Date("2020-12-31T00:00:00.000Z"))).toBe(false);
  });
});

We've caught this test and fixed it, but there are still other tests with the same issue waiting to blow up. People are also forgetful - we'll write more tests that pass now but won't pass the next day/week/month/year, so what do we do?

Random Time as a Default

We can make use of randomization to get our tests to fail now instead of later. If we default time as a random time that changes per run, we can take advantage of this randomization to detect tests that rely on a certain real-world time. This also helps the team build good habits of addressing time in tests.

// jest.setup.js
function randomDate(start, end) {
  // Source: https://stackoverflow.com/questions/9035627/elegant-method-to-generate-array-of-random-dates-within-two-dates
  return new Date(
    start.getTime() + Math.random() * (end.getTime() - start.getTime())
  );
}

beforeEach(() => {
  // setup our fake timers
  jest.useFakeTimers("modern");
  jest.setSystemTime(
    randomDate(
      new Date("2000-01-01T00:00:00.000Z"),
      new Date("2040-01-01T00:00:00.000Z")
    )
  );
});

afterEach(() => {
  // cleanup fake timers
  jest.runOnlyPendingTimers();
  jest.useRealTimers();
});

Note: I chose Jan 1, 2000, to Jan 1, 2040, as an appropriate range for my application. You should pick the range that best fits your needs.

You can add this snippet to your global jest setup. After several runs, it will identify any tests that implicitly rely on time.

Catching Issues Sooner

You should make sure that this check can flag issues before they merge. For example, you could add redundancy to test runs before PR merge by running them multiple times in CI.

Conclusion

Fake timers are great, but enforcing their use can be tricky. Time can be deeply embedded in our applications in ways we don't immediately see. Take advantage of randomization to root out implicit assumptions, and make them explicit so your tests can withstand the test of time.

Appendix: Full Setup

You can take a look at the whole setup with some more tests in this example GitHub repo, or browse this repl.it:

Sorry, repl.it embeds don't let me choose the default file. Check out jest.setup.js and relies-on-time.test.js.