DEV Community

Lucas Pereira de Souza
Lucas Pereira de Souza

Posted on

Mutation Testing with Stryker

#typescript #node #testing

logotech

## Mutants in the Code: How Stryker Mutator Boosts Your Backend Quality

Have you ever stopped to think about how reliable your tests truly are? In a world where software quality is increasingly crucial, especially in backend systems, blindly trusting code coverage can be a trap. This is where mutant testing comes in – a powerful technique to go beyond the surface and ensure your tests actually catch the bugs that matter. In this post, we'll demystify mutant testing, learn how to set up Stryker Mutator in a TypeScript/Node.js project, and most importantly, interpret the generated reports to elevate your code quality.

The Problem: The False Sense of Security from Code Coverage

Automated tests are the backbone of any robust software project. They give us the confidence to refactor, add new features, and generally maintain sanity amidst growing complexity. However, the traditional code coverage metric, while useful, can be misleading. A test might \"cover\" a line of code without actually verifying its behavior correctly. This means a subtle but critical change could go unnoticed, leading to production bugs.

Imagine a test that checks if a number is greater than 10. If the condition is changed to \"greater than 5,\" code coverage might remain at 100%, but the test would fail to detect an error if the expected value was actually 8. Mutant testing emerges as a solution to this problem.

Mutant Testing: Introducing the Enemy to Conquer

Mutant testing is a technique that automatically modifies small parts of your source code (creating \"mutants\") and then runs your existing tests to see if they detect these changes. If a test fails after a modification, the mutant is considered \"killed,\" indicating that your test was effective in catching that change. If no tests fail, the mutant \"survives,\" signaling a potential weakness in your test suite.

The process generally follows these steps:

  1. Mutant Generation: Tools like Stryker Mutator introduce small syntactic changes to your code (e.g., replacing > with <, + with -, true with false, removing return, etc.).
  2. Test Execution: Your existing tests are run against each modified version of the code (each mutant).
  3. Results Analysis:
    • Killed Mutant: If at least one test fails, the mutant is considered killed. This is good, as it means your tests caught the alteration.
    • Survived Mutant: If all tests pass, the mutant survived. This is a warning sign, indicating your tests are not sensitive enough to that specific modification.

Setting Up Stryker Mutator with TypeScript/Node.js

Let's get hands-on! We'll assume you already have a Node.js project with TypeScript and a configured test suite (e.g., with Jest).

1. Installation:

First, install Stryker Mutator and the necessary plugins as development dependencies:

npm install --save-dev @stryker-mutator/core @stryker-mutator/jest-runner @stryker-mutator/typescript
Enter fullscreen mode Exit fullscreen mode

2. Initial Configuration (stryker.conf.json):

Create a stryker.conf.json file in the root of your project. This file instructs Stryker on how to operate.

// stryker.conf.json
{
  \"$schema\": \"./node_modules/@stryker-mutator/core/schema/stryker-schema.json\",
  \"mutate\": [
    \"src/**/*.ts\" // Path to the files you want to mutate (e.g., all source code in 'src')
  ],
  \"testRunner\": \"jest\", // The test runner you are using
  \"reporters\": [
    \"clear-text\", // Readable report in the console
    \"html\" // Interactive HTML report
  ],
  \"htmlReporter\": {
    \"baseDir\": \"stryker-report\" // Directory where the HTML report will be generated
  },
  \"jest\": {
    // Jest-specific configurations, if needed.
    // For example, to specify the Jest config file:
    // \"config\": \"jest.config.js\"
  },
  \"thresholds\": {
    \"high\": 80, // Mutants killed % above which the build is considered successful
    \"low\": 60,  // Mutants killed % above which the build is considered acceptable
    \"break\": 60 // Mutants killed % below which the build should break
  },
  \"plugins\": [
    \"@stryker-mutator/jest-runner\",
    \"@stryker-mutator/typescript\"
  ],
  \"tempDirName\": \".temp-stryker\", // Temporary directory for mutants
  \"cleanTempDir\": true // Clean the temporary directory after execution
}
Enter fullscreen mode Exit fullscreen mode

Explanation of Key Fields:

  • mutate: Defines which files and directories Stryker should analyze to create mutants.
  • testRunner: Informs Stryker which test runner to use (Jest, Mocha, etc.).
  • reporters: Specifies the report formats. clear-text is great for the console, html generates an interactive report.
  • htmlReporter: Configures the output directory for the HTML report.
  • jest: Allows passing additional configurations to Jest.
  • thresholds: Sets percentage targets for the killed mutant rate. break is crucial for CI/CD as it can fail the build if the target isn't met.
  • plugins: Loads the necessary plugins for your test runner and language.

3. Running Stryker:

In your terminal, execute the command:

npx stryker run
Enter fullscreen mode Exit fullscreen mode

Stryker will:

  • Read your configuration.
  • Identify files to mutate.
  • Generate mutants for each file.
  • Run your tests against each mutant.
  • Generate reports as configured.

Example Code and Tests (TypeScript/Node.js):

Let's consider a simple example of a calculation function in a file src/calculator.ts:

// src/calculator.ts

/**
 * Adds two numbers.
 * @param a The first number.
 * @param b The second number.
 * @returns The sum of a and b.
 */
export function add(a: number, b: number): number {
  // If a mutant changes '+' to '-', this test should kill it.
  return a + b;
}

/**
 * Checks if a number is positive.
 * @param num The number to check.
 * @returns True if the number is positive, False otherwise.
 */
export function isPositive(num: number): boolean {
  // If a mutant changes '>=' to '>', this test should kill it.
  return num >= 0;
}
Enter fullscreen mode Exit fullscreen mode

And its corresponding tests in src/calculator.test.ts (using Jest):

// src/calculator.test.ts
import { add, isPositive } from './calculator';

describe('Calculator', () => {
  describe('add', () => {
    it('should return the sum of two positive numbers', () => {
      expect(add(2, 3)).toBe(5);
    });

    it('should return the sum of a positive and a negative number', () => {
      expect(add(5, -2)).toBe(3);
    });

    it('should return the sum when one number is zero', () => {
      expect(add(0, 7)).toBe(7);
    });

    // Test to ensure the mutant changing '+' to '-' is caught
    it('should correctly add negative numbers', () => {
      expect(add(-1, -2)).toBe(-3);
    });
  });

  describe('isPositive', () => {
    it('should return true for a positive number', () => {
      expect(isPositive(10)).toBe(true);
    });

    it('should return true for zero', () => {
      // If a mutant changes '>=' to '>', this test should kill it.
      expect(isPositive(0)).toBe(true);
    });

    it('should return false for a negative number', () => {
      expect(isPositive(-5)).toBe(false);
    });
  });
});
Enter fullscreen mode Exit fullscreen mode

If you run npx stryker run with this code, Stryker will create mutants such as:

  • return a - b; (in the add function)
  • return a * b; (in the add function)
  • return num > 0; (in the isPositive function)

Your current tests are good enough to kill most of these mutants, but Stryker will tell us exactly which ones survived.

Interpreting the Mutation Report

After execution, Stryker will present a report in the console and, if configured, an interactive HTML report.

Console Report (Simplified Example):

Stryker: Running tests...
Stryker: Detected 2 files to mutate.
Stryker: Running tests against 2 mutants.
... (progress) ...
Stryker: All tests passed on the original code.

----------------------------------- Mutant test results ----------------------------------
| Status        | Count | Percentage | Description                                              |
|---------------|-------|------------|----------------------------------------------------------|
| Killed        | 15    | 75%        | Tests failed on this mutant.                             |
| Alive         | 5     | 25%        | All tests passed on this mutant.                         |
| Pending       | 0     | 0%         | Not tested due to timeout or other reasons.              |
| Ignored       | 0     | 0%         | Mutant was ignored based on stryker configuration.       |
| Total         | 20    | 100%       |                                                          |
------------------------------------------------------------------------------------------
Mutation Score: 75%
Enter fullscreen mode Exit fullscreen mode

Key Points for Interpretation:

  • Mutation Score: The most important metric. It's the percentage of mutants that were \"killed\" by your tests. A high score indicates your tests are effective at detecting alterations.
  • Killed vs. Alive:
    • Killed: Great! Your tests detected the failure introduced by the mutant.
    • Alive: Red flag! Your tests didn't fail even with the code alteration. This suggests you need to add or improve your tests to cover that specific scenario.
  • HTML Report: The HTML report (generated in the stryker-report directory by default) is extremely useful. It offers a detailed view:
    • Lists all mutants.
    • Shows the status of each (Killed, Alive, etc.).
    • Allows you to click on an \"Alive" mutant to see exactly which part of the code was modified and which tests passed. This directs your efforts where they are most needed.
    • Visualizes test coverage at the mutant level.

What to Do with Survived Mutants?

  1. Improve Your Tests: The most common scenario is that an alive mutant indicates a flaw in your test suite. You need to add assertions or test cases that cover the condition altered by the mutant. For example, if the mutant return num > 0; survived in our isPositive function, it means no test verified the exact behavior when the number is positive (and not zero). Adding a test like expect(isPositive(5)).toBe(true); would kill that mutant.
  2. Adjust Stryker's Configuration: In some rare cases, an alive mutant might be acceptable. Perhaps the mutation introduces a trivial bug or a condition that should never occur in production. You can configure Stryker to ignore certain files or specific mutations using ignorePatterns in stryker.conf.json. Use this cautiously!

Conclusion: Towards More Reliable Code

Mutant testing, especially with tools like Stryker Mutator, is a natural and necessary evolution when striving for software quality excellence. It forces us to think critically about our tests and ensure they don't just run, but validate the correct behavior of our code.

By integrating Stryker into your development workflow and CI/CD, you add a robust layer of validation, significantly increasing confidence in your codebase and preventing subtle bugs that could otherwise slip through. Don't settle for code coverage; embrace mutant testing and build truly robust and reliable backends.

Top comments (0)