DEV Community: Jessica Sales Melo

Mutation Testing: Why Code Coverage Isn't Enough

Jessica Sales Melo — Thu, 03 Sep 2026 13:48:06 +0000

Mutation Testing: Why Code Coverage Isn't Enough

How the Stryker tool reveals the true strength of your test suite — using a real business rule with clear limits.

A green "100% code coverage" badge feels great. But here's the uncomfortable question QA should ask: coverage says a line ran — does it say the behaviour was verified? If someone changed that line, would your tests notice?

Mutation testing answers exactly that. In this article I show it with Stryker + Jest, applied to a grain-service-fee business rule (a nod to agribusiness domain rules).

Full, runnable source on my GitHub.

What is a mutant?

Stryker makes a tiny change to your production code — a mutant — and re-runs the tests:

tests fail → mutant killed ✅ (your test detected the change)
tests pass → mutant survived ❌ (that line is executed, but not truly verified)

mutation score = killed mutants / total mutants

The killer example: 100% coverage vs ~86% mutation score

Two modules both report 100% line, statement and branch coverage. Only mutation testing tells them apart:

Module	Coverage	Mutation score (weak suite)
Leap-year check	100%	100% (strong)
Taxa-service-fee rule	100%	86% — 5 mutants survive 🚨

The fee rule is a classic off-by-one factory:

if (value <= 1000) return 0;            // isento (exempt)
const rate = type === 'soja' ? 0.02 : 0.015;
return round(value * rate);

The 5 surviving mutants are business bugs waiting to ship

Surviving mutant	Why it survives
`value <= 1000` → `value < 1000`	the exact value 1000 is never tested
`value < 0` → `value <= 0`	the value 0 is never tested
`typeof value !== 'number'` → `false`	no test passes a non-numeric value
error message changed to `''`	tests check `toThrow(Type)` but not the message

These are off-by-one boundary bugs — precisely the defects that reach production wearing a green coverage badge.

The fix: boundary + message tests

Adding tests on the exact boundaries (0, 1000, non-numeric input) and asserting error messages kills every mutant → 100% mutation score.

Configuring the quality gate

Stryker can fail the build just like a failing test:

"thresholds": { "high": 95, "low": 80, "break": 95 }

With break: 95, a weak suite exits 1; a strong one exits 0. Wire this into CI and you turn coverage into an effectiveness gate.

Cost & levers

Each mutant re-runs the suite — slow. Mitigations: coverageAnalysis: "perTest" (only tests touching the line run), --incremental, and mutate only changed files in CI.

Equivalent mutants (the honest caveat)

A mutation that can't change observable behaviour (e.g. a pure logging call) is un-killable — that's why 100% isn't always the right goal. Use Stryker's thresholds and // Stryker disable comments.

Takeaways

Coverage ≠ quality. Mutation testing measures your assertions, not your execution.
Surviving mutants are a to-do list: missing test cases, weak assertions, or dead code to delete.
Best where logic is dense in boundaries: pricing, validation, permissions, business rules.

The demo (strong vs weak suite, the SOLUTION=1 toggle, and the CI quality gate) is on my GitHub — a great conversation starter for senior QA roles.

Jessica Sales — QA Engineer. When I say my tests are good, mutation testing is what lets me prove it.

AI-Powered Test Case Generation: Turning a Feature Description into Executable Gherkin

Jessica Sales Melo — Thu, 03 Sep 2026 13:42:36 +0000

AI-Powered Test Case Generation: Turning a Feature Description into Executable Gherkin

A practical look at plugging an LLM into the QA cycle — and why a controlled vocabulary keeps AI-generated tests runnable.

As a QA Engineer, one of my biggest experiments this year was: can AI write tests that actually run? The short answer is yes — if you constrain the "creativity." In this article I share a working approach where you describe a feature in plain text and an AI generator produces Gherkin scenarios executed by Playwright.

This is a write-up of a portfolio project. The full, runnable source is on my GitHub.

The pipeline

feature description (.md)  →  🧠 AI generator  →  Gherkin (.feature)  →  Playwright runs it

1. The goal: text → runnable Gherkin

The user writes a natural-language spec (what the feature does and its acceptance criteria). The generator uses AI to produce a .feature file. Then playwright-bdd runs it for real.

2. The key design decision: a pluggable IAProvider

The heart is a simple interface:

interface IAProvider {
  generateFeature(spec: string, glossary: string): Promise<string>;
}

Two implementations:

Provider	Used when	What it does
`deterministic`	No API key	Generates scenarios from curated templates, detected by keywords in the spec. Runs on any environment (local and CI), no secret needed.
`openai`	`OPENAI_API_KEY` set	Uses a real LLM to derive new scenarios from the description.

Selection is automatic: if a key exists, use the LLM; otherwise fall back to deterministic. This keeps the pipeline run-ready even with no credentials.

3. The trick that makes AI tests executable: a controlled glossary

This is the most important lesson. If you let an LLM invent any step text, you end up with beautiful scenarios that cannot run — because no step definition matches them.

Solution: give the model a glossary of supported steps and ask it to generate only within that vocabulary:

PASSOS DISPONÍVEIS (use exactly these):
- Given I am on the page "PAGINA"
- When I check checkbox number N
- When I select the option "OPÇÃO" in the dropdown
...

The prompt also includes app context (credentials, expected messages). Result: AI-generated Gherkin always matches the step definitions.

4. Prompt engineering for deterministic-enough output

temperature: 0.3 → low creativity, high adherence.
System prompt: "Generate ONLY the Gherkin file, starting with # language: pt. Do not add commentary."
Post-process: strip markdown fences the model sometimes wraps in.

5. Validation

bddgen (missing-step detection) runs as a quality gate — if the AI invents a step, generation fails early instead of at runtime. Then Playwright executes the scenarios.

Takeaways

AI accelerates authoring, but the executable contract (glossary ↔ step definitions) is what keeps quality high.
A pluggable provider makes the tool work everywhere: deterministic in CI, real LLM locally.
This is a realistic template for "AI in QA" that isn't just a demo — it produces tests that run.

The source (with the deterministic provider, OpenAI provider, and the glossary) is public on my GitHub — a good starting point if you want to experiment with AI-assisted test generation.

Jessica Sales — QA Engineer, exploring AI applied to the test automation lifecycle.

Playwright + BDD: Building an End-to-End Test Automation Framework from Scratch

Jessica Sales Melo — Thu, 03 Sep 2026 13:42:35 +0000

Playwright + BDD: Building an End-to-End Test Automation Framework from Scratch

A practical guide to building a maintainable E2E test suite with Playwright, TypeScript and Gherkin.

Why this article? As a QA Engineer, I often hear that test automation is "just clicking buttons with code." In this post I walk through a production-ready approach: a BDD-driven E2E framework that is readable by devs/POs/QAs, easy to maintain, and green in CI — built with Playwright, TypeScript and playwright-bdd (Gherkin in Portuguese, the Belgium of the project's origin, but the pattern applies to any language).

This is a write-up of a portfolio project — the full, runnable source is on GitHub.

The stack

Tool	Why
Playwright (1.6x)	Reliable cross-browser automation (Chromium + Firefox here)
TypeScript (strict)	Type safety across page objects and steps
playwright-bdd (9.x)	Writes scenarios in Gherkin and executes them with the Playwright runner
Page Objects (POM)	Centralized selectors, no hardcoding in tests
Allure Report	Rich report with steps, screenshots and history
GitHub Actions	CI that runs the suite and publishes the report on GitHub Pages

Project structure

├── features/            # BDD scenarios (Gherkin)
│   ├── login.feature
│   ├── inventory.feature
│   └── checkout.feature
├── steps/               # Step definitions (bridge Gherkin to code)
├── fixtures/            # Extended fixtures + injected Page Objects
├── src/pages/           # Page Objects
├── support/             # Environment config (.env)
└── playwright.config.ts # Playwright + playwright-bdd config

1. BDD-first: write the feature, then the code

The beauty of BDD is the conversation. Scenarios are written in plain language, so Product Owners, Devs and QAs share the same understanding:

Feature: Login
  Scenario: Login with valid credentials
    Given I am on the login page
    When I fill in the valid credentials
    Then I am taken to the products page

2. Bind Gherkin to code with step definitions

Each Given/When/Then maps to a step definition that uses fixtures to inject Page Objects:

Given('I am on the login page', async ({ loginPage }) => {
  await loginPage.open();
});

When('I fill in the valid credentials', async ({ loginPage }) => {
  await loginPage.login(STANDARD_USER, PASSWORD);
});

Then('I am taken to the products page', async ({ inventoryPage }) => {
  await inventoryPage.expectVisible();
});

3. Page Objects: keep selectors in one place

A Page Object encapsulates the page's elements and actions, so the framework stays maintainable when the UI changes:

export class LoginPage {
  constructor(readonly page: Page) {
    this.username = page.locator('#user-name');
    this.password = page.locator('#password');
    this.submit = page.locator('#login-button');
  }

  async login(user: string, pass: string) {
    await this.page.goto('/');
    await this.username.fill(user);
    await this.password.fill(pass);
    await this.submit.click();
  }
}

4. Pluggable fixtures make steps concise

Using test.extend() + createBdd(), the Page Objects are injected into every step — no globals, no setup duplication.

5. Report & CI

Allure gives a rich, visual report; the CI publishes it on GitHub Pages (a live, navigable link in your README).
Robustness is a feature: workers: 1 + navigation retries tame flaky network access to public demo apps — the same lesson applies to single-session systems.

Takeaways

BDD is a collaboration tool, not just syntax.
Page Objects + typed fixtures = the difference between a script that breaks and a framework that scales.
Green CI with published reports is what makes a portfolio credible.

Want the full code? Check out the open-source project linked on my GitHub, where all 7 scenarios run green in CI with an Allure report deployed to GitHub Pages.

Jessica Sales — QA Engineer, focused on end-to-end automation, API testing and applying AI to the QA lifecycle.