Math.random() in tests

Let's assume following source code generating random number with Math.random() and faker.random.number():

var faker = require('faker');

faker.seed(1);

console.log(
  "Faker goes:", 
  faker.random.number({ max: 100 })
);

console.log(
  "Math.random() goes:", 
  Math.random() * 100
);

And now let's try to run it several times in a row:

❯ node index.js
Faker goes: 42
Math.random() goes: 24.270154608840078

❯ node index.js
Faker goes: 42
Math.random() goes: 17.379030134115037

❯ node index.js
Faker goes: 42
Math.random() goes: 66.8433058100395

One quite common thing to do while writing tests is to write down what data got generated during the test run and assert against that value:

let myRandomInt = faker.random.number({ max: 100 });
// let myRandomInt = Math.random() * 100;

assert.equals(myRandomInt, 42, 'Coincidence? I think not');

Now while there are better methods to write tests, this is a quick win and when done right can work quite well. As you can see from the values generated above when you use Math.random() you will get different result every time. While when using faker the results seem stable.

Faker will always give you the same results when all the calls to it are exactly the same till your call. The problem arises when you for some reason add another call to faker before your call:

faker.random.number(); // Extra faker call

let myRandomInt = faker.random.number({ max: 100 });
assert.equals( // This will fail ☹️
  myRandomInt, 
  42, 
  'Coincidence? I think not'
);

How to solve this? Using seed, which will reset the pseudo-random sequence:

faker.random.number(); // Extra faker call

faker.seed(1);
let myRandomInt = faker.random.number({ max: 100 });
assert.equals( // It works again ✨
  myRandomInt, 
  42, 
  'Coincidence? I think not'
);

Conclusion

If you want to hard-code assert expected values in your test, you should make sure that you:

1. In your tests use stable random generator like faker instead Math.random().
2. Pin faker.seed(x) to a constant value before generating data for each test.

---

Photo by Riho Kroll on Unsplash

Comments

[Elliot Nelson]

Good summary! Curious, under what circumstances would you want to generate seeded random values in a test?

(As opposed to just explicitly stubbing Math.random() to return 43, for example.)

[Michal Bryxí]

Great question. I would say that you actually always want to have a pseudo-random seed (good trick is to pin it to pull-request branch name), because this gives you the ability to "shake the universe". Which turns out tend to uncover a lot of edge-cases your manual dice roll did not account for. On each run of the CI you are more and more sure that every possible input combination is taken care of without writing additional test code.

I would go as far as stating that an ember app (that's what I mostly work with) of any non-trivial size/complexity that does not use pseudo-random seed in the tests for running ember-exam has at least one hidden race condition.

I had few posts touching this topic (1), (2)