DEV Community

Parsa
Parsa

Posted on AI-assisted

green tests don't cover the branch that leaks

#ai #testing #python #postgres

agent-written tests mostly test the path the agent wrote, which is the success path. so the bugs that survive are the ones that only run when something fails.

the cleanest example going around

IBM ContextForge, issue #1885. the session cleanup looked like this

finally:
    db.commit()   # can raise
    db.close()    # never reached if it does
Enter fullscreen mode Exit fullscreen mode

finally guarantees the block starts, not that every line in it finishes. if commit() throws, python leaves the block right there and close() never runs.

every test was green. under a 3,000 user load test, postgres showed this

        state        | count
---------------------+-------
 idle in transaction |    81
 idle                |    20
 active              |     4
Enter fullscreen mode Exit fullscreen mode

81 connections checked out, doing nothing, until postgres killed them at 120 seconds. the fix (PR #1886) nests the commit so the close always runs

finally:
    try:
        db.commit()
    finally:
        db.close()
Enter fullscreen mode Exit fullscreen mode

nothing in the original diff looks wrong. the leak lives in the exit path nobody exercised.

catching this before load does

force the failure branch in a test. mock the commit to raise, then assert the connection went back to the pool.

# pytest + pytest-mock, adjust the import paths to your app
import pytest
from app.auth import check_team_membership

def test_session_closed_when_commit_fails(mocker):
    session = mocker.MagicMock()
    session.commit.side_effect = RuntimeError("commit failed")
    mocker.patch("app.db.SessionLocal", return_value=session)

    with pytest.raises(RuntimeError):
        check_team_membership("user-1")

    session.close.assert_called_once()
Enter fullscreen mode Exit fullscreen mode

grep for cleanup that depends on an earlier call succeeding. commit then close in one finally is the classic, but any acquire, do work, release with no nested try has the same shape. a with SessionLocal() as db: block avoids it entirely.

watch postgres during any load test.

SELECT state, count(*)
FROM pg_stat_activity
WHERE datname = current_database()
GROUP BY state;
Enter fullscreen mode Exit fullscreen mode

idle in transaction climbing while traffic stays flat means something isn't releasing.

the blind test

the ContextForge case got written up as a blind scan. the pre-fix file went through Tomosu cold, with no issue, no PR and no hint, against a success criterion fixed before the scan, and it flagged the unsafe cleanup. full write-up with the method and its limits in the case study.

what's the leak that took you longest to find?

Top comments (0)