DEV Community

TECHDEV
TECHDEV

Posted on Originally published at github.com

Your CI is green. Your tests might be lying.

#testing #ai #python #javascript

Your CI is green. Your tests might be lying.

I shipped a bug last week that my test suite had "covered." The suite was
green. The test just wasn't testing anything anymore.

That is the failure mode nobody warns you about: a suite can go green
without a single line of production code improving. An AI coding agent
under pressure — or a tired human at 11pm — can make a failing test pass
without fixing the code:

  • delete the failing test,
  • wrap the assertion in a retry loop (for _ in range(5): try: assert ...),
  • swallow the AssertionError in try/except: pass,
  • @pytest.mark.skip the flaky test,
  • assert True,
  • time.sleep(2) and hope the race goes away.

Every one of those leaves CI green. Every one of those hides a real bug.
So I built a small tool that scans for exactly those patterns:
testguard.

What it checks

One stdlib-only Python CLI. No dependencies, no network, no telemetry.
Python checks run on the real AST, not regexes, so formatting tricks
don't hide a fake pass. JS/TS checks are line-based and cover Jest,
Vitest, Mocha, Bun, and Node's built-in runner.

python -m testguard tests/
Enter fullscreen mode Exit fullscreen mode
Check Severity Meaning
empty-test high Test body is only pass / docstring — can never fail
fake-pass high assert True, expect(true).toBe(true) — can never fail
retry-loop high Assertion inside a retry loop — passes if it succeeds once
swallowed-assert high Assertion in try/except: pass — failure is discarded
deleted-test high (diff mode) a test was deleted or renamed
removed-assert high (diff mode) assertion removed, no replacement added
focused-test high .only() — runs one test, silently skips the rest
no-assert medium Test runs code but asserts nothing
hardcoded-sleep medium time.sleep / setTimeout hides timing flakiness
mock-only medium Test asserts only on mocks, not on real behaviour
skipped-test low @pytest.mark.skip, test.skip, .todo — never runs
expected-fail low xfail without strict=True — failure never blocks CI

The part that matters most: diff mode

Scanning the whole suite is useful. Scanning the change is where it
pays off — a PR that weakens tests gets flagged before it merges:

testguard --diff HEAD
testguard --diff main
Enter fullscreen mode Exit fullscreen mode

A deleted test, a removed assertion, or a freshly added assert True
in a PR is almost always worth a second look in review.

Demo 

$ python -m testguard tests/fixtures/py_cheating.py
testguard: 10 suspicious pattern(s) found:
    31  [high  ] retry-loop: Assertion inside a retry loop: the test passes if it succeeds once.
    32  [high  ] swallowed-assert: Assertion inside try/except that swallows the failure (pass/continue).
    47  [low   ] skipped-test: Test is skipped and never runs.
    ...

$ python -m testguard tests/fixtures/py_clean.py
testguard: clean — no suspicious test patterns found.
Enter fullscreen mode Exit fullscreen mode

Use it as a CI gate 

# Only fail the build on high-severity findings
testguard tests/ --fail-on high

# Report only, never fail
testguard tests/ --fail-on none
Enter fullscreen mode Exit fullscreen mode 
- name: Fail on manipulated tests
  run: python -m testguard tests/ --fail-on medium
Enter fullscreen mode Exit fullscreen mode

One honesty note

testguard reports patterns, not guilt. A retry loop around a genuinely
flaky third-party call, a skipped test with a linked issue, or a mock-heavy
test for a thin adapter can all be legitimate. Treat each finding as
"look here and decide", not "this code is cheating". That's why
--fail-on exists, and why the report always shows its evidence.

Green CI is a signal. It is not proof. If an agent — or a tired human —
touched your tests, check that the tests still test something.

Repo: https://github.com/techdevfun/testguard — MIT, zero dependencies,
Python 3.9+.

Top comments (0)