When the AI Agent Blamed the Database: A Debugging Retrospective

At 2:14 AM on a Tuesday that I will not miss, a background worker that had processed jobs quietly for three weeks started dying. The dashboard showed connection reset by peer on every retry, and the on-call page fired so many times that I silenced it out of self-preservation. Like most of us under pressure, I pasted the trace into an AI assistant and asked for a verdict.

The model, running through MonkeyCode's free model access — which, as of this writing, advertises a 10-million-token allowance on its free tier — was confident: the database was exhausted, the pool was too small, and I should raise max_connections and tune idle_in_transaction_session_timeout. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I spent two hours resizing pools and restarting Postgres, and the job failed again at 4:47 AM with the same reset.

The database was fine, and the bug was in my own code, which is the part the AI politely skipped. To be clear, this incident is reconstructed and compressed from a failure pattern that shows up in worker code everywhere; the repro below is minimal, runnable, and deliberately small. That night taught me a workflow I now apply to every late-night incident: an AI-generated root cause is a hypothesis, not a verdict, and you reproduce before you theorize.

I took the worker, pointed it at a disposable Postgres instance on the free server option that MonkeyCode provides, and fed it a single record that matched the failure window. It died in four minutes instead of forty, which is the whole point of a repro environment: it compresses the incident into something you can watch. The third rule is to trust the trace, not the explanation, because the worker's retry loop looked innocent enough:

for attempt in range(5):
    try:
        process(record)
        break
    except Exception:
        logger.warning("retrying %s (attempt %d)", record["id"], attempt)

And process looked fine too, until you noticed the early return:

def process(record):
    conn = pool.getconn()
    cur = conn.cursor()
    cur.execute("SELECT payload FROM jobs WHERE id = %s", (record["id"],))
    row = cur.fetchone()
    if row is None or row[0] is None:
        return  # the connection never goes back to the pool
    handle(row[0])
    pool.putconn(conn)

A record with a null payload hit the early return, leaked a connection, and the broad except Exception retried the same record five times, leaking five connections per job. After a few hundred jobs the pool was empty, Postgres started resetting connections, and the sanitized warning log hid the leak behind a network error. The AI's diagnosis was wrong because it only saw the final symptom, exactly like a doctor diagnosing a fever without asking about the mosquito bite.

The fix was small, but the discipline around it matters more. Return the connection in a finally block, catch only the exceptions you can actually retry, and always preserve the original traceback in the log:

def process(record):
    conn = pool.getconn()
    try:
        cur = conn.cursor()
        cur.execute("SELECT payload FROM jobs WHERE id = %s", (record["id"],))
        row = cur.fetchone()
        if row is None or row[0] is None:
            return
        handle(row[0])
    finally:
        pool.putconn(conn)

The retry loop now catches OperationalError specifically, re-raises everything else, and logs the full exception chain so the next incident does not require a midnight archaeology session. The fourth rule is to bisect your changes before you blame your infrastructure, so I ran git bisect start, marked the last known good deploy, and let the repro test decide which commit introduced the null-payload branch:

git bisect start
git bisect bad
git bisect good 3f2a9c1
git bisect run pytest tests/test_worker.py

It found the culprit in six steps: a defensive null check added two weeks earlier that nobody realized skipped the cleanup path. That commit was supposed to make the worker more robust, and it instead made it leak on every null payload. The lesson is that every early return is a contract with your resources, and the AI is very good at writing contracts but very bad at noticing the ones you already signed.

Where the free model access genuinely helped was instrumentation, not diagnosis. I asked it to write a script that sampled open connections while the worker ran, and it produced a ten-line watcher that showed the pool draining in real time:

watch -n 1 "psql -c \"SELECT count(*) FROM pg_stat_activity WHERE datname = 'worker'\""

Seeing the count climb with every retry turned the incident from a mystery into a graph, and that is the kind of task where a language model earns its keep. It also explained pool semantics patiently at 5 AM, which is more than I can say for most documentation.

Now the honest limitations, because this workflow is not universal. If your failure only appears under production traffic, a free server with a toy dataset will not reproduce it, and you need real load or a traffic replay instead. If you are chasing memory corruption, kernel behavior, or hardware faults, an LLM's guesses are close to useless, and you should reach for profilers and core dumps first. Treat the free server as a scratch environment, not a production host, the same way you would treat any free tier: it is for compressing incidents, not for hosting them.

The thing that changed for me is the order of operations. Reproduce first, instrument second, let the AI write the instrumentation, and treat every root-cause claim as a suspect until the repro proves it. If you want to try the same loop, MonkeyCode's docs walk through connecting the free model access and spinning up the free server, and the rest is just the discipline of not trusting the first confident answer.