My Vue Library Had 99% Line Coverage. Mutation Testing Found Two Bugs Anyway.

On October 2nd, vue-smart-loading-kit had 1,369 unit tests and 99% line coverage. That afternoon I added Stryker for mutation testing. About an hour and a half later I released 0.3.1 with fixes for two bugs my tests had missed.

What mutation testing checks

Coverage tells you which lines ran during your tests. It can't tell you whether a test would fail if one of those lines were wrong.

Stryker checks that part. It makes one small change to your source at a time, like flipping a condition, changing a default or removing a call, and runs your tests against each changed version. Each version is called a mutant. If a test fails, the mutant is killed. If all your tests still pass, it survived, and you've found code that no test really checks.

The config

I run mutation testing against the unit tests only. My real-browser tests are too slow to rerun for every mutant.

{
  "testRunner": "vitest",
  "vitest": { "configFile": "vitest.stryker.config.ts" },
  "mutate": ["src/**/*.{ts,vue}", "!src/**/*.d.ts", "!src/**/types.ts", "!src/**/*.types.ts"],
  "coverageAnalysis": "perTest",
  "ignoreStatic": true,
  "incremental": true,
  "plugins": ["@stryker-mutator/vitest-runner", "./scripts/stryker-vue-macros.mjs"],
  "ignorers": ["vue-macros"],
  "thresholds": { "high": 90, "low": 80, "break": null }
}

vitest.stryker.config.ts is my unit test config with tests/browser/** excluded. With incremental on, Stryker saves its results to a file, and the next run only retests mutants touched by changed code or tests. In CI I cache that file between runs, so a small commit doesn't mean a full run.

Workaround 1: Vitest 5 skips every test

With @stryker-mutator/vitest-runner 10.0.0 and Vitest 5, every mutant survives. The report makes your tests look useless, and none of them actually ran.

For each mutant, the runner picks the tests to run with a name pattern. It builds that pattern by joining suite and test names with a space. Vitest 5 joins them with " > ", so the pattern matches nothing and Vitest skips everything.

I run this script before each mutation run. It patches the join in the runner's two files:

// scripts/stryker-vitest5-fix.mjs
import { readFileSync, writeFileSync } from 'node:fs'

const dir = 'node_modules/@stryker-mutator/vitest-runner/dist/src'
const before = "return nameParts.join(' ').trim();"
const after = "return nameParts.filter(Boolean).join(' > ').trim();"

for (const file of ['test-helpers.js', 'stryker-setup.js']) {
  const path = `${dir}/${file}`
  const source = readFileSync(path, 'utf8')
  if (source.includes(after)) continue
  if (!source.includes(before)) {
    throw new Error(`${path}: collectTestName changed; check whether this workaround is still needed`)
  }
  writeFileSync(path, source.replace(before, after))
}

If the runner's code changes, the script throws, so I'll find out when I can delete it. That was the state of things in early October. Check your runner version before you copy this.

Workaround 2: Vue's compiler macros

When Vue compiles a <script setup> component, it moves the arguments of defineProps, withDefaults and the other macros out of setup(). A mutant in there calls Stryker's helper functions before they exist, and the component fails to compile.

An ignorer plugin tells Stryker to leave those calls alone:

// scripts/stryker-vue-macros.mjs
import { PluginKind, declareValuePlugin } from '@stryker-mutator/api/plugin'

const MACROS = new Set(['defineProps', 'withDefaults', 'defineEmits', 'defineModel', 'defineSlots', 'defineOptions'])

export const strykerPlugins = [
  declareValuePlugin(PluginKind.Ignore, 'vue-macros', {
    shouldIgnore(path) {
      if (path.isCallExpression() && MACROS.has(path.node.callee.name)) {
        return 'Vue compiler macro: arguments are hoisted out of setup()'
      }
    },
  }),
]

The prop defaults still have tests. The component tests cover them.

Bug 1: #12345z parsed as a color

The skeleton works out its highlight color from your base color, so the library has a small color parser. Here's the old hex part, shortened:

function parseHex(hex: string): RGBA | null {
  const h = hex.replace('#', '').trim()
  if (![3, 4, 6, 8].includes(h.length)) return null
  // ...
  r = parseInt(h.slice(0, 2), 16)
  g = parseInt(h.slice(2, 4), 16)
  b = parseInt(h.slice(4, 6), 16)
  // ...
  if ([r, g, b, a].some((n) => Number.isNaN(n))) return null
  return { r, g, b, a: a / 255 }
}

parseInt reads digits until it hits a character it can't use, then returns what it has so far:

parseInt('zz', 16) // NaN
parseInt('5z', 16) // 5

So the NaN check rejected #zz1234 and let #12345z through as { r: 18, g: 52, b: 5, a: 1 }. In a six- or eight-digit color, it only caught a bad character when it came first in its pair.

In fairness to coverage, the warning was there if I had looked per file. color.ts was one of my weakest files at 75% of statements, and the return null lines for bad input had never run. I was looking at the 99% total.

The fix checks the whole string up front:

if (![3, 4, 6, 8].includes(h.length) || !/^[0-9a-f]+$/i.test(h)) return null

After that, parseInt can't return NaN here, so I deleted the old check. A mutant inside a condition that can never be true has no test that could kill it.

Bug 2: aria-hidden="false" disappeared after loading

This one was in a file with 100% line coverage.

While v-skeleton is showing, it sets inert, aria-hidden and aria-busy on your element. When loading ends, it puts back what your template bound. The old code treated false as "remove the attribute":

if (v === undefined || v === null || v === false) return [name, null]
if (v === true) return [name, name === 'inert' ? '' : 'true']
return [name, String(v)]

Vue 3 doesn't remove an aria-* attribute bound to false. It renders the string "false":

<section v-skeleton="loading" :aria-hidden="false">
  ...
</section>

Before loading, that element had aria-hidden="false". After loading, the attribute was gone. The line with v === false ran in my tests, but no test bound an aria attribute to false and checked what came back.

The fix copies Vue's behavior. inert is the exception, because Vue sets it as a DOM property instead of an attribute:

if (v === undefined || v === null) return [name, null]
if (name === 'inert' && typeof v === 'boolean') return [name, v ? '' : null]
return [name, String(v)]

If you run the tests from that fix against the old code, both fail with exactly these values: the parser returns { r: 18, g: 52, b: 5, a: 1 } for #12345z, and aria-hidden comes back as null instead of "false".

After the fixes

The mutation score is over 90%. It's a live badge in the README, and the full report is published with the demo, so you can see which mutants still survive. I also raised the coverage thresholds in CI the same day, to 98% of statements and 99% of lines.

The score doesn't fail the build (break: null). Some mutants don't change what the code does at all, and no test can kill those.

For a library this size, the setup was one config file and two short scripts, and it found real bugs on the first afternoon.

If you try it

Read the list of surviving mutants before you look at the score. For me, the survivors pointed at files that the total coverage number made look fine.

• npm: vue-smart-loading-kit
• GitHub: https://github.com/navidjaberi/vue-smart-loading-kit
• Mutation report: https://navidjaberi.github.io/vue-smart-loading-kit/mutation/
• More about v-skeleton: I built a Vue directive that turns your real component into its own skeleton
• I also write about Vue and frontend on LinkedIn

If you've tried mutation testing on a Vue or React project, what did the survivors show you?