So, what is mutation testing? Well, it is a type of testing which allows us to evaluate the quality of our tests.
Of course, we could check the code coverage to see if our tests execute all our source code. With that, we could think we are testing all the possibilities and be confident that we don't have any bugs, right?
So let's take a look at this little example:
function compareGreaterThan18(a) {
return a > 18;
}
Here we can see a simple function, which returns true if the function's parameter is bigger than 18 and false otherwise.
Let's set up our test runner web-test-runner
-
Install web-test-runner:
npm i --save-dev @web/test-runner -
Install chai:
npm i --save-dev @esm-bundle/chai -
Create the configuration for wtr (although it can also be executed with just web-test-runner paht/to/*.test.js --node-resolve)
Just create a
web-test-runner.config.mjsfile in the root of your project:
export default { coverage: true, files: ['./src/test/*.test.js'], nodeResolve: true, rootDir: '../../', // }rootDir is used to resolve modules in a monorepo, in this case, we need to set it up so Stryker can resolve the modules correctly.
You can check all the options at https://modern-web.dev/docs/test-runner/cli-and-configuration/
-
Now, we can create our test:
import { expect } from '@esm-bundle/chai'; import { compareGreaterThan18 } from '../compareGreaterThan18.js' describe('compareGreaterThan18', () => { it('should return true if the number is greater than 18', () => { expect(compareGreaterThan18(27)).to.be.true; }); }); -
Execute the test
npx wtr
And with that, we got a 100% of code coverage but are we sure that this test is enough?
No, it is not enough. What happens if someone changes our > inside our code to >=?... Well, the test will still work when it should have failed.
And the same happens if the 18 is changed to another number lower than 27.
In this example, it is easy to see what tests should have added, but it is not always that easy to see what changes in our code could add bugs, and we wouldn't notice it because the tests say everything is ok.
So now, let's see how we can solve this.
Let's set up Stryker Mutator
Stryker is a JavaScript mutation testing framework.
It will modify your code by adding some mutants. For example, in the previous function, it will change the > to >= or it will change it to <.
Then if your tests fail, the mutant is killed, but otherwise, it means the mutant survived, which can indicate that we have not tested everything should have tested.
So let's kill some mutants.
-
Install Stryker
npm i --save-dev @stryker-mutator/core -
Create the configuration for Stryker
The file is called
stryker.conf.js
/** * @type {import('@stryker-mutator/api/core').StrykerOptions} */ module.exports = { testRunner: 'command', files: ['src/*.js', 'src/**/*.test.js', 'package.json', '*.mjs'], mutate: ['src/*.js', '!src/**/*.test.js'], packageManager: 'npm', reporters: ['html', 'clear-text', 'progress'], };Here we set up our test runner in this case it will be command as we just want to execute our test command which will be
npm test.With the
filesproperty, you can choose which files should be included in the test runner sandbox and normally you don't need to set it up because by default it uses all the files not ignored by git.And then we add the files we want to mutate
'src/*.js'and the ones we don't want to mutate'!src/**/*.test.js'to the array mutate.All the options can be checked at https://stryker-mutator.io/docs/stryker/configuration
-
Set your test command to execute wtr
"scripts": { "test": "wtr" }, -
Modify our web test runner config so it works together with Stryker
Stryker uses mutation switching to be able to put all the mutants into the code simultaneously, this way it doesn't need to modify your code before running each mutation.
Then it uses an environment variable to select which mutation is being tested
__STRYKER_ACTIVE_MUTANT__.With web-test-runner we are running the tests in a browser so we have to inject this variable so the tests can read and use it.
In our
web-test-runner.config.mjswe set the testRunnerHtml property to inject active mutant:
function getCurrentMutant() { return process.env.__STRYKER_ACTIVE_MUTANT__; } export default { coverage: true, files: ['./src/test/*.test.js'], nodeResolve: true, rootDir: '../../', testRunnerHtml: testFramework => `<html> <body> <script> window.__stryker__ = window.__stryker__ || {}; window.__stryker__.activeMutant = ${getCurrentMutant()}; window.process = { env: { __STRYKER_ACTIVE_MUTANT__: ${getCurrentMutant()}, } } </script> <script type="module" src="${testFramework}"></script> </body> </html>`, }From the version 5 and onwards of Stryker the
__STRYKER_ACTIVE_MUTANT__andactiveMutantmust be of type String so be sure to put double quotes or single quotes around the expression${getCurrentMutant()}.
window.__stryker__ = window.__stryker__ || {}; window.__stryker__.activeMutant = '${getCurrentMutant()}'; // Single quotes to be sure it is a string so it works on Stryker version 5 window.process = { env: { __STRYKER_ACTIVE_MUTANT__: '${getCurrentMutant()}', // Single quotes to be sure it is a string so it works on Stryker version 5 } } -
Now, we can run our mutation testing
npx stryker runOnce it finishes, we will see a report like this one:
In this case, we can see that our test was not able to survive 2 mutants out of 5.
So now let's kill some mutants!
Let's add some tests to kill the mutants
The first survived mutant is the following one:
- return a > 18;
+ return true;
The minus symbol indicates what was changed and the plus indicates what was changed.
Here we can see that if our statement was to be changed to always return true, our test would still say that everything is ok which should not be the case and it could be the origin of bugs in the future.
So let's fix it, we have to add a test where we check what happens if a is lower than 18.
it('should return true if the number is greater than 18', () => {
expect(compareGreaterThan18(14)).to.be.false;
});
With this test, we have killed one mutant and we can kill the one left.
- return a > 18;
+ return a >= 18;
This mutant is sowing us that we don't check what happens if a is 18 and we don't have any test checking it so we have to add one:
it('should return true if the number is greater than 18', () => {
expect(compareGreaterThan18(18)).to.be.false;
});
And... congratulations, now we have killed all the mutants!!!!!
Conclusion
With this, we were able to see that code coverage doesn't tell us if our tests are good or bad, instead, we should execute mutation tests as we did with Stryker.
One way to be more confident about our tests, for example, is to check the score calculated by Stryker, the higher the score the more confident we can be about our tests.
And mutation testing can take a lot of time, in the example showed it only takes 3 seconds to execute all the tests, but as your project grows, it will take a lot more.
- Only mutate what you need to mutate, don't mutate your demo folders or your mocks.
- Try to improve the performance of your tests: run tests concurrently, load just what you need to run the tests, stub functions you should not tests, etc
Top comments (2)
Really great stuff! Thank you for sharing this. It helped me figure out why my mutation tests weren’t working. 💙
Side note: the ‘package.json’ you’ve included in the files causes issues. :)
I'm glad it was helpful 😁.
What issue did you have with the 'package.json'? Did removing it solve the issue?