DEV Community

SVS_Praveen
SVS_Praveen

Posted on

Why 100% Line Coverage is Lying to You in AI-Generated Code (And How We Catch It)

#ai #python #testing #opensource

If you have spent the last few months building projects with AI coding assistants (Antigravity, Claude Code, Cursor, Copilot), you have likely experienced this specific frustration:

You prompt an agent to build a feature or fix a bug. The agent writes tests. You run pytest, and all green checkmarks appear with 100% line coverage. You feel confident and push to production — only to discover after deployment that the tests were completely hollow and missed critical edge-case logic.

Line coverage measures whether a line of code was executed, not whether its logic was actually asserted.

To solve this, I built and open-sourced DeployProof — a deterministic pre-push verification tool for Python that catches hollow test suites, hallucinated dependencies, and security traps in seconds before code leaves your local machine.

The Illusion of Green Line Coverage

To illustrate the problem clearly, consider this simple discount calculator with a 50% threshold cap:

# calculator.py
def calculate_discount(price: float, rate: float) -> float:
    if rate > 0.5:
        return price * 0.5
    return price * (1.0 - rate)
Enter fullscreen mode Exit fullscreen mode

When asked to write unit tests, an LLM might generate this:

# test_calculator.py
from calculator import calculate_discount

def test_calculate_discount_standard():
    assert calculate_discount(100.0, 0.2) == 80.0
Enter fullscreen mode Exit fullscreen mode

This single test hits every branch of the standard discount and yields 100% line coverage.

However, if you mutate the logic:

  • Change rate > 0.5 to rate > 1.5
  • Change return price * 0.5 to return price * 1.5
  • Change * to /

The test suite still passes 100% green. The test never asserted the threshold cap or boundary conditions.

Why Existing Mutation Testing Was Too Slow

Traditional mutation testing tools (like mutmut or cosmic-ray) are powerful, but they typically run against the entire codebase. On a project with hundreds of tests, running a full mutation suite can take 5 to 20 minutes — far too slow to run on every git commit or pre-push hook.

DeployProof solves this with Diff-Scoped AST Mutation:
Instead of mutating the entire repository, DeployProof inspects your active git diff (or uncommitted session files) and targets AST mutations strictly to the lines you just wrote or modified.

This drops verification time from minutes down to 2 to 4 seconds.

$ deployproof check

DeployProof - LOCAL PRE-CHECK
====================================================================
Target Scope (1 file evaluated):
  * calculator.py

Local Pre-Check Mutation Verification:
  Score:  57.1% (4/7 mutants killed)
  Status: FAILED (score 57.1% below 80.0%) (threshold: 80.0%)
  Time:   2.27s

Surviving Mutants (3 unverified changes):
  [1] calculator.py:2
      Mutation: Replace numeric constant '0.5' with '1.5'
      Original: if rate > 0.5:
      Mutated:  if rate > 1.5:

  [2] calculator.py:3
      Mutation: Replace numeric constant '0.5' with '1.5'
      Original: return price * 0.5
      Mutated:  return price * 1.5

  [3] calculator.py:3
      Mutation: Replace binary operator '*' with '/'
      Original: return price * 0.5
      Mutated:  return price / 0.5
====================================================================
Pre-check FAILED: Score 57.1% is below threshold 80.0% (3 surviving mutants).
Enter fullscreen mode Exit fullscreen mode

Once you add tests for the threshold cap (rate = 0.8) and exact boundary (rate = 0.5), all mutants are killed and the pre-push gate passes at 100.0%.

5 Additional Verification Passes

Beyond hollow tests, AI codebases frequently introduce adjacent failure modes. DeployProof runs 5 additional static verification passes against your active diff:

  1. PyPI Dependency & Slopsquatting Defense: Queries the live PyPI registry to verify every newly imported module exists, protecting against hallucinated package names.
  2. GhostApproval Symlink Traps (CWE-61): Catches symlinks pointing outside the repository root designed to escape developer sandboxes.
  3. Control Flow & Error Handling: Flags empty except Exception: pass blocks and dead code generated to silence errors.
  4. Mock-Introduction Auditing: Flags newly introduced @patch and unittest.mock usage that masks broken business logic.
  5. Credential Scanner: Catches unquoted .env secrets and hardcoded API keys (OpenAI, Anthropic, AWS, Stripe).

Quickstart

DeployProof is free, open source (MIT), and installs via pip:

pip install deployproof
Enter fullscreen mode Exit fullscreen mode

Initialize it in your repository (creates .deployproof.json and sets up the .git/hooks/pre-push gate to block pushes when checks fail):

deployproof init
Enter fullscreen mode Exit fullscreen mode

Run on-demand verification anytime:

deployproof check
Enter fullscreen mode Exit fullscreen mode

For CI/CD pipelines (GitHub Actions, GitLab CI), it provides structured JSON output:

deployproof check --json
Enter fullscreen mode Exit fullscreen mode

Links & Contributing

I built DeployProof as an independent solo developer after repeatedly hitting subtle AI test regressions across my own projects.

If you are using AI coding agents in your daily workflow, I would love for you to try it out, file issues, star the repository, or contribute:

What subtle failure modes or hollow test patterns have you noticed in your AI coding workflows? Let me know in the comments below!

Top comments (3)

Collapse
 
reidmarlow profile image
Reid Marlow •

Diff-scoping AST mutations to the active diff is the only way mutation testing ever survives a pre-push hook. Full-repo mutmut runs take long enough that people disable the git hook after two days.

The trickiest edge case with agent-written diffs is when the agent only modifies test_*.py to make a run pass by weakening assertions, leaving the implementation untouched. If the diff has no implementation lines, pure diff-scoped mutation needs to trace imports back to the target module or it only mutates the test harness itself.

Collapse
 
svspraveen profile image
SVS_Praveen •

Thanks Reid! Totally agree on hook speed — whenever a hook takes more than a few seconds, people just end up using git commit --no-verify and the safety is gone.

And you hit the nail on the head with the test-only diffs. I've seen agents do exactly that: when a test fails, instead of fixing the logic, they just weaken the assert or patch the function out to get a green checkmark.

Right now DeployProof flags newly introduced mocks/patches and lets you pass --files to target the implementation, but having it automatically trace test_*.py imports back to the source module is a killer idea. Definitely adding that to the roadmap.

Thanks for taking the time to read and share this, really appreciate it!

Collapse
 
svspraveen profile image
SVS_Praveen •

Thanks for reading! If anyone tries running deployproof check on their local projects or wants to test against specific agent frameworks, I'd love to hear how it performs or if you hit any edge cases.