Four High-severity bugs were hiding behind a green test suite in a 7k-star library

Jeff Lenamon — Fri, 31 Jul 2026 13:23:34 +0000

Run pytest on kennethreitz/records at upstream HEAD and you get the answer every maintainer wants: 31 passed. Green across the board.

At that exact same commit, all of the following are true:

db.query("INSERT ...") silently loses your data. The rows vanish when the connection closes. No error, no warning, nothing.
db.bulk_query(...) loses data the same way.
db.transaction() swallows every exception raised inside it. Your failed transaction reports success.
Every single query() call leaks a pooled connection. With a default-sized pool, the third query can hang your process.

Four High-severity bugs. One green test suite. I want to walk through how that is possible, because the mechanism is more interesting than any single bug, and it is probably present in more codebases than we would like to admit.

Full disclosure before we start: I found these with an autonomous audit loop I built, and I will say a little about it at the end. Everything in this post is independently checkable. The repro script runs against upstream HEAD, and all four bugs were disclosed upstream with a PR offer in records#236. None of this is a criticism of records or its maintainers. It is a beloved library with a beautiful API, and what happened to it is a story about dependency semantics, not carelessness.

What actually broke

records was written in the SQLAlchemy 1.x era, and its Connection class leans on two 1.x behaviors that SQLAlchemy 2.0 removed.

First, autocommit. Under 1.x, executing a bare INSERT through a connection could commit implicitly. Under 2.x, autocommit is gone: if nobody calls commit(), the transaction rolls back when the connection closes. records never calls commit() for plain queries, because it never had to. So on SQLAlchemy 2.x, db.query("INSERT INTO users ...") executes, appears to succeed, and the row quietly disappears when the connection goes away. Same story for bulk_query.

Second, close_with_result. Under 1.x, this flag meant the connection would close itself once the result set was exhausted, which is how records returned connections to the pool. Under 2.x the flag is accepted and ignored. Every query() checks out a pooled connection and never returns it. A pool of size 1 with 1 overflow dies on the third query.

The third bug is older and more human. db.transaction() wraps its body in a bare except: that rolls back and then does not re-raise. Callers cannot tell a failed transaction from a successful one. Here is the detail that stopped me: upstream added the missing raise on 2026-02-08 and reverted it the same day. The fix existed for a few hours. Something in the test suite presumably objected, and the fix lost.

Which brings us to the real subject.

Why the suite stayed green

The records test suite runs against SQLite, in memory. That choice is fast, hermetic, and completely blind to all four bugs.

An in-memory SQLite database lives on a single connection. There is no separate reader that would fail to see uncommitted rows, so the missing commits are invisible: within that one shared connection, uncommitted state looks exactly like committed state. There is no real pool boundary to exhaust, so the leaked connections are invisible too. The suite cannot observe persistence, cannot observe pool behavior, and cannot observe commit semantics, because in its world those concepts do not exist.

And the transaction bug? One test actively asserted the broken behavior. The suite did not just miss the bug. It defended it. That is almost certainly why the one-day fix got reverted: the fix made a test fail, and the test was wrong.

This is the generalizable lesson, and it has nothing to do with AI or with records specifically. A test gate is a measurement instrument, and an instrument can be precise, fast, reliable, and pointed at the wrong world. records' gate measured "does this code work against a single shared in-memory connection," and the answer was honestly yes. Nobody was asking "does this code work against a database," which is the only question users care about.

Green is not a property of the code. Green is a property of the code and the gate together.

How the bugs were found

I ran an autonomous improvement loop against a local clone. The loop's first iteration is always an audit, and the audit has one hard rule: a finding exists only if it can be reproduced on the spot. No "this looks wrong," no linting vibes. Every claim above began life as a small script the audit ran against upstream HEAD and watched fail.

The audit also noticed something structural: three of the four Highs are the same bug wearing different costumes. Missing commits, ignored close_with_result, and the varargs breakage all trace to one root cause, a Connection class written against SQLAlchemy 1.x execution semantics. The loop has a three-strike rule for exactly this pattern: the third finding sharing one root cause forces a single structural fix, never a third spot patch.

So the fix was structural. Commit-and-finalize when a result set is exhausted, immediate commit for DML statements, single-result connections actually returned to the pool, and a transaction wrapper that guarantees auto-commit never fires inside an explicit transaction. That last clause matters: the naive spot patch (just commit after every statement) silently breaks rollback, because a rollback that follows an auto-commit rolls back nothing. The structural fix proves rollback still works, with a test.

The transaction() fix restores the change upstream made and reverted: rollback, then re-raise.

Giving the gate teeth

Fixing code under a blind gate is pointless, so the loop's next task rebuilt the instrument. The test that asserted the swallow bug was corrected. A file-backed regression suite was added: persistence through both query paths and both API forms, pool exhaustion, exception propagation, explicit rollback.

Then the honesty check, which is my favorite artifact of the whole run: the new regression suite was executed against the pre-fix code from the audit checkpoint, and 5 of its 6 tests fail there. A regression test that never failed on the broken code proves nothing. This one is proven to detect the world it claims to detect. After the fixes, the full suite is 37 passing.

What it declined to do

Two things did not happen, and they matter as much as the fixes.

The loop declined to impose a lint gate, with a written reason: records declares no lint configuration, so a lint pass would be cosmetic churn on a dormant project. And two decisions it had no right to make were filed for the owner instead of taken: whether to replace the unmaintained docopt dependency, and whether to restore the multi-backend CI matrix. An autonomous tool that seizes owner decisions is worse than no tool.

Everything shipped as artifacts: a repro.py that demonstrates all four bugs on upstream HEAD and their absence after the patch, a fixes.patch of about 180 lines covering code, tests, and packaging, and a journal of every iteration including the operator mistakes the method itself caught (a premature "passed" claim, an invalid negative-path check, and one mistyped commit hash in a ledger, each caught and corrected by the loop's own rules).

All four findings went upstream with the repros and a PR offer. Merging anything is, as it should be, the maintainers' call.

Takeaways

A green suite certifies the gate, not the code. Ask what world your gate can actually observe. In-memory databases, mocked networks, and frozen clocks are all worlds with physics different from production.
When a correct fix makes a test fail, suspect the test. The one-day life of the raise fix is what it looks like when a wrong test wins.
Regression tests should be proven against the broken code. If you never watched them fail, you do not know what they detect.
The third bug with the same root cause is not a bug, it is architecture. Fix the boundary, not the symptom.

The tool, briefly

The loop is called Jeffy Loop. It is a free, MIT-licensed autonomous improvement loop for Claude Code: audit, backlog with a runnable acceptance check per task, one verified task per iteration behind local git checkpoints and a verify gate, convergence only when a fresh audit comes back clean. The records run used 7 of its 8-iteration budget. The whole engine is one shell script of about 100 lines, and the eval above, alongside the rest of the eval set across eight languages including deliberate control cases that came back clean, lives in the repo with every artifact mentioned here: https://github.com/lenamonj/jeffy-loop

DEV Community: Jeff Lenamon