DEV Community: SVS_Praveen

Why 100% Line Coverage is a Dangerous Lie (And How We Made Python Mutation Testing Run in Seconds)

SVS_Praveen — Thu, 03 Sep 2026 17:34:39 +0000

We've all seen pull requests boasting 90%+ or even 100% line coverage. Everything looks green, the test suite passes in CI, and the PR gets merged.

A few days later, a subtle logic bug blows up production.

How does this happen? Because line coverage measures execution paths, not assertion quality.

With the explosion of AI coding assistants (Copilot, Cursor, Claude, ChatGPT), generating tests has become effortless. But AI assistants routinely generate boilerplate tests that execute functions without asserting true invariants:

def test_calculate_discount():
    # Executes every line in calculate_discount(), yielding 100% line coverage!
    result = calculate_discount(price=100, is_vip=True)
    assert result is not None  # Never asserts the actual discount math!

That test passes with flying colors while touching 100% of the function's lines. But if a bug inverts price * 0.8 to price * 1.5, the test still passes and the bug ships unnoticed.

🎬 See DeployProof in Action

The Gold Standard: Mutation Testing

The true metric of test suite integrity is mutation testing:

An engine modifies your code’s Abstract Syntax Tree (AST) — swapping == to !=, < to >=, inverting arithmetic (* to /), or replacing constants (0.5 to 1.5).
It runs your test suite against each generated "mutant."
If your tests fail, the mutant is killed (your tests assert true correctness).
If your tests pass, the mutant survived (your test coverage is hollow).

The Bottleneck: Why Nobody Used Mutation Testing

Traditional mutation testing tools like mutmut or Cosmic Ray rewrite files to disk and re-test your entire repository. On a codebase with hundreds of mutants, running test suites repeatedly takes 20 to 60+ minutes.

Because of that latency, mutation testing remained an expensive, rarely run overnight CI job rather than an active pre-push quality gate.

The Solution: DeployProof and In-Memory AST Mutation

I built DeployProof to eliminate the latency bottleneck.

Instead of modifying files on disk, DeployProof:

Scopes directly to your git diff: Only newly written or modified lines in your active session are evaluated.
In-Memory AST Schemata: Injects all AST mutants into a unified compiled tree switched dynamically in warm Python interpreter memory (__DEPLOYPROOF_MUTANT__), completely bypassing disk I/O.
Dead-Code & Equivalence Pruning: Static taint analyzer skips unkillable dead code and equivalent mutants before test dispatch.

By combining in-memory AST schemata with diff-scoping, DeployProof drops the feedback loop down to 2 to 4 seconds. You get instant, deterministic proof of whether your new code has genuine assertion backing before you push:

$ deployproof check

Target Scope (1 file evaluated):
  * src/auth/tokens.py

Local Pre-Check Mutation Verification:
  Score:  100.0% (6/6 mutants killed)
  Status: PASSED (0 surviving mutants) (threshold: 80.0%)
  Time:   1.84s

[Gate Result] PASSED (All 7 Verification Gates Clean)

🧬 Feature Highlight: Actionable Self-Healing Tests

When a mutant survives because a test is missing an assertion, DeployProof doesn't just give you an error — it can write the fix for you:

deployproof check --heal-tests

DeployProof analyzes the surviving AST mutation, infers parameter types, and auto-synthesizes a ready-to-run pytest test case:

# Auto-generated by DeployProof in tests/test_deployproof_healed.py
def test_kill_calculate_discount_line_4():
    """
    Auto-synthesized test to kill surviving mutant on line 4.
    Target:  'if is_vip: return price * 0.8'
    Mutated: 'if is_vip: return price * 1.5'
    """
    from auth.tokens import calculate_discount
    result = calculate_discount(price=100, is_vip=True)
    assert result == 80.0

You can even run in Interactive Mode (deployproof check -i) to review and apply synthesized tests with single-keystroke confirmation.

🔒 The 7 Deterministic Verification Gates

In addition to in-memory mutation testing, DeployProof evaluates every change against 6 other critical hygiene gates before code can leave your machine:

🧬 In-Memory AST Mutation Engine: Swaps operators, boundary values, and return statements in warm memory.
✨ Self-Healing Test Synthesizer: Generates copy-pasteable pytest cases to close assertion gaps.
🔒 OWASP Top 10 SAST Scanner: Detects SQL injection, shell command execution, insecure deserialization (pickle), and path traversals via AST visitors.
🔑 Shannon Entropy Secrets Scanner: Scans working tree files and up to 50 previous git commits to catch hardcoded API keys, tokens, and tracked .env files.
📦 OSV.dev Live CVE & Slopsquatting Defense: Queries OSV.dev for known dependency advisories and PyPI to catch hallucinated package names invented by LLMs.
🔗 GhostApproval Symlink Defense: Traps repository sandbox-escaping symlinks before commits reach CI.
⚙️ Control Flow & Strict Error Handling: Detects bare except:, swallowed exceptions (except Exception: pass), and unreachable dead code.

⚡ Cross-Platform & Windows WSL Acceleration

DeployProof runs natively on macOS, Linux, and Windows. On Windows, you can optionally pass --wsl to seamlessly delegate execution to the native Linux kernel for maximum speed with automatic Windows-to-POSIX path mapping.

5-Minute Quickstart

DeployProof is free, open-source (MIT licensed), and available on PyPI.

Installation

# Recommended: Global CLI install via pipx
pipx install deployproof

# Or via standard pip
pip install deployproof

Essential Commands

# Run pre-push gate on current git diff (2–4s)
deployproof check

# Synthesize self-healing tests for surviving mutants
deployproof check --heal-tests

# Install 1-click pre-push git hook
deployproof init

# Audit entire repository with multi-worker sandboxes
deployproof check --full-repo --workers 8

# Output machine-readable JSON for CI pipelines & IDEs
deployproof check --json

Privacy & Zero Telemetry Guarantee

DeployProof runs 100% locally on your machine.

Zero external telemetry or analytics.
Zero cloud dependencies or accounts required.
The only outbound network query is a read-only call to the official PyPI registry / OSV database to verify package safety.

Try It & Share Feedback

DeployProof has been validated against 278+ test cases and architectural patterns from 250+ popular open-source repositories.

🐙 GitHub: https://github.com/SVSPraveen/deployproof
📦 PyPI: https://pypi.org/project/deployproof/
📖 Documentation Portal: https://svspraveen.github.io/deployproof/

If you give it a spin on your repositories, let me know what your mutation score looks like in the comments! ⭐

Why 100% Test Coverage is Deceptive (And How We Made Python Mutation Testing Fast Enough for Pre-Push)

SVS_Praveen — Tue, 01 Sep 2026 17:19:05 +0000

We've all seen pull requests boasting 90%+ or even 100% line coverage. Everything looks green, the tests pass in CI, and the PR gets merged.

A few days later, a subtle logic bug blows up production.

How does this happen? Because line coverage measures execution paths, not assertion quality.

With the explosion of AI coding assistants (Copilot, Cursor, Claude), generating tests has become trivial. But AI assistants routinely generate boilerplate tests that execute functions without asserting true invariants:

def test_calculate_discount():
    # Executes every line in calculate_discount(), giving 100% line coverage!
    result = calculate_discount(price=100, is_vip=True)
    assert result is not None  # Never asserts the actual discount math!

That test passes with flying colors while touching 100% of the function's lines. But if a bug inverts price * 0.8 to price * 1.5, the test still passes!

The Gold Standard: Mutation Testing

The true metric of test suite integrity is mutation testing:

An engine modifies your code’s Abstract Syntax Tree (AST) — swapping == to !=, < to >=, or replacing return values with None.
It runs your test suite against each generated "mutant."
If your tests fail, the mutant is killed (your tests assert true correctness).
If your tests pass, the mutant survived (your test coverage is hollow).

The Bottleneck: Mutation Testing is Painfully Slow

Traditional mutation testing tools like mutmut test your entire repository. On a codebase with hundreds of mutants, running test suites repeatedly takes 20 to 60+ minutes.

Because of that latency, mutation testing has remained an expensive, rarely run overnight CI job rather than an active pre-commit or pre-push quality gate.

The Solution: DeployProof and Diff-Scoped AST Mutation

I built DeployProof to solve the latency problem.

Instead of mutating your entire codebase, DeployProof:

Parses your active git diff against your base branch or uncommitted working tree.
Translates the modified line spans into their specific AST subtrees.
Generates and executes isolated mutations strictly on the newly written or modified logic.

By scoping mutations directly to touched code, DeployProof drops the feedback loop down to 2 to 5 seconds. You get instant, deterministic proof of whether your new code has genuine assertion backing before you push:

$ deployproof check

[Target Discovery]
  Target Files: 1 (src/auth/tokens.py)
  Test Files:   1 (tests/test_tokens.py)

[Mutation Engine] (diff-scoped)
  Generated Mutants: 6
  Killed:            6
  Survived:          0
  Mutation Score:    100.0%

[Gate Result] PASSED (duration: 2.14s)

4 Other Pre-Push Hygiene Gates

While building the diff-scoped AST engine, I added 4 critical sanity checks that standard linters miss:

Dependency Hallucination & Slopsquatting Defense: AI coding tools frequently introduce dependencies that do not exist on public PyPI. DeployProof queries the official PyPI JSON API before push to verify every new package in requirements.txt or inline imports actually exists.
Shannon Entropy Secret Scanning: Catches accidentally tracked .env files, leaked OpenAI/Anthropic/AWS API keys, and bearer tokens introduced in session diffs using entropy analysis.
Control Flow & Swallowed Exceptions: Detects and blocks dangerous blanket except Exception: pass anti-patterns that hide runtime failures, as well as unverified mock fixtures.
GhostApproval Symlink Defense: Traps repository sandbox-escaping symlinks before commits can reach CI.

Deep Multi-Worker Audits

If you want to audit your entire repository, DeployProof includes a parallel multi-worker engine:

deployproof check --full-repo --workers 8

It uses a ProcessPoolExecutor where each worker runs in an isolated, PID-keyed filesystem sandbox (worker_<PID>) with dedicated --override-ini=cache_dir=... and separate --basetemp=... pytest roots to scale across CPU cores without lock contention or state leaks.

Privacy & Zero Telemetry

DeployProof runs 100% locally on your machine.

There is zero telemetry, zero analytics, and zero external servers. The only outbound network call it makes is querying the public PyPI JSON API to check if a newly imported package exists.

Getting Started

DeployProof is free, open-source (MIT licensed), and available on PyPI.

Installation

# Recommended: Isolated global CLI install
pipx install deployproof

# Or via standard pip
pip install deployproof

Run Checks

# Check active git diff (2–5s)
deployproof check

# Output machine-readable JSON for CI/CD pipelines
deployproof check --json

Try It & Share Feedback

DeployProof has been verified against major open-source repositories including requests, click, and colorama.

GitHub: [https://github.com/SVSPraveen/DeployProof](https://github.com/SVSPraveen/DeployProof)
PyPI: https://pypi.org/project/deployproof/

If you test it out on your repositories, drop a comment below with your thoughts, edge cases, or feature requests! If you find it useful, a star on GitHub is always appreciated! ⭐

Why 100% Line Coverage is Lying to You in AI-Generated Code (And How We Catch It)

SVS_Praveen — Sat, 29 Aug 2026 14:15:45 +0000

If you have spent the last few months building projects with AI coding assistants (Antigravity, Claude Code, Cursor, Copilot), you have likely experienced this specific frustration:

You prompt an agent to build a feature or fix a bug. The agent writes tests. You run pytest, and all green checkmarks appear with 100% line coverage. You feel confident and push to production — only to discover after deployment that the tests were completely hollow and missed critical edge-case logic.

Line coverage measures whether a line of code was executed, not whether its logic was actually asserted.

To solve this, I built and open-sourced DeployProof — a deterministic pre-push verification tool for Python that catches hollow test suites, hallucinated dependencies, and security traps in seconds before code leaves your local machine.

The Illusion of Green Line Coverage

To illustrate the problem clearly, consider this simple discount calculator with a 50% threshold cap:

# calculator.py
def calculate_discount(price: float, rate: float) -> float:
    if rate > 0.5:
        return price * 0.5
    return price * (1.0 - rate)

When asked to write unit tests, an LLM might generate this:

# test_calculator.py
from calculator import calculate_discount

def test_calculate_discount_standard():
    assert calculate_discount(100.0, 0.2) == 80.0

This single test hits every branch of the standard discount and yields 100% line coverage.

However, if you mutate the logic:

Change rate > 0.5 to rate > 1.5
Change return price * 0.5 to return price * 1.5
Change * to /

The test suite still passes 100% green. The test never asserted the threshold cap or boundary conditions.

Why Existing Mutation Testing Was Too Slow

Traditional mutation testing tools (like mutmut or cosmic-ray) are powerful, but they typically run against the entire codebase. On a project with hundreds of tests, running a full mutation suite can take 5 to 20 minutes — far too slow to run on every git commit or pre-push hook.

DeployProof solves this with Diff-Scoped AST Mutation:
Instead of mutating the entire repository, DeployProof inspects your active git diff (or uncommitted session files) and targets AST mutations strictly to the lines you just wrote or modified.

This drops verification time from minutes down to 2 to 4 seconds.

$ deployproof check

DeployProof - LOCAL PRE-CHECK
====================================================================
Target Scope (1 file evaluated):
  * calculator.py

Local Pre-Check Mutation Verification:
  Score:  57.1% (4/7 mutants killed)
  Status: FAILED (score 57.1% below 80.0%) (threshold: 80.0%)
  Time:   2.27s

Surviving Mutants (3 unverified changes):
  [1] calculator.py:2
      Mutation: Replace numeric constant '0.5' with '1.5'
      Original: if rate > 0.5:
      Mutated:  if rate > 1.5:

  [2] calculator.py:3
      Mutation: Replace numeric constant '0.5' with '1.5'
      Original: return price * 0.5
      Mutated:  return price * 1.5

  [3] calculator.py:3
      Mutation: Replace binary operator '*' with '/'
      Original: return price * 0.5
      Mutated:  return price / 0.5
====================================================================
Pre-check FAILED: Score 57.1% is below threshold 80.0% (3 surviving mutants).

Once you add tests for the threshold cap (rate = 0.8) and exact boundary (rate = 0.5), all mutants are killed and the pre-push gate passes at 100.0%.

5 Additional Verification Passes

Beyond hollow tests, AI codebases frequently introduce adjacent failure modes. DeployProof runs 5 additional static verification passes against your active diff:

PyPI Dependency & Slopsquatting Defense: Queries the live PyPI registry to verify every newly imported module exists, protecting against hallucinated package names.
GhostApproval Symlink Traps (CWE-61): Catches symlinks pointing outside the repository root designed to escape developer sandboxes.
Control Flow & Error Handling: Flags empty except Exception: pass blocks and dead code generated to silence errors.
Mock-Introduction Auditing: Flags newly introduced @patch and unittest.mock usage that masks broken business logic.
Credential Scanner: Catches unquoted .env secrets and hardcoded API keys (OpenAI, Anthropic, AWS, Stripe).

Quickstart

DeployProof is free, open source (MIT), and installs via pip:

pip install deployproof

Initialize it in your repository (creates .deployproof.json and sets up the .git/hooks/pre-push gate to block pushes when checks fail):

deployproof init

Run on-demand verification anytime:

deployproof check

For CI/CD pipelines (GitHub Actions, GitLab CI), it provides structured JSON output:

deployproof check --json

Exit Codes

0 — PASSED: All verification checks passed.
1 — FAILED: Code quality or security gate triggered (mutation score below threshold, leaked secrets, fake dependencies, symlink escape).
2 — ERROR: Test environment failure (test suite failed to collect before mutation testing began).

Links & Contributing

I built DeployProof as an independent solo developer after repeatedly hitting subtle AI test regressions across my own projects.

If you are using AI coding agents in your daily workflow, I would love for you to try it out, file issues, star the repository, or contribute:

💻 GitHub (MIT): https://github.com/SVSPraveen/DeployProof
📦 PyPI: https://pypi.org/project/deployproof/
🧪 Verified Test Suite: 88/88 pytest unit tests and 11/11 launch-day stress test fixtures reproducing each planted edge case.

What subtle failure modes or hollow test patterns have you noticed in your AI coding workflows? Let me know in the comments below!