DEV Community: Parsa Mohammadi

A Blind Scan Found IBM ContextForge's Connection Leak From One File

Parsa Mohammadi — Wed, 21 Oct 2026 15:31:45 +0000

On January 3, 2026, a 3,000-user load test against IBM ContextForge left PostgreSQL connections stuck in idle in transaction until the pool ran dry. The cause was three lines of cleanup code. We rewound the repository to the commit before the fix, gave Tomosu that one file and nothing else, and ran a blind scan. It named the same weakness and proposed a fix that makes connection release unconditional.

42 to 81	120 s	27 / 100	Direct hit
connections stuck `idle in transaction`	held before a timeout released them	PRI on the blind scan	against a criterion sealed before scanning

The system

ContextForge is an open-source gateway and registry for MCP, A2A, REST and gRPC. Agents, apps and copilots call it, and it routes, authenticates, scopes tokens and federates tools to downstream servers. Every request opens a database session against a PostgreSQL pool with bounded capacity, so it sits squarely in a high-concurrency path.

What the load test found

The team ran 3,000 concurrent users to surface behavior ordinary development never hits. Requests looked healthy, but connections stopped cycling. Between 42 and 81 sat in idle in transaction: checked out, doing nothing, unavailable to anyone else. Each one stayed stuck until PostgreSQL's idle_in_transaction_session_timeout killed it at 120 seconds. Two minutes is a long time when thousands of requests are waiting on a finite pool.

The stuck query was a SELECT from the token-scoping middleware (issue #1885).

Three lines of cleanup

The investigation led to mcpgateway/middleware/token_scoping.py. It looks fine at a glance: a finally, a commit, a close.

finally:
    db.commit()   # can raise
    db.close()    # never reached if commit raises

finally guarantees the block starts, not that every statement in it finishes. db.commit() can fail (a PgBouncer timeout, a connection reset). When it raises, Python leaves the block and close() never runs. Under enough concurrency, three lines become pool exhaustion.

The issue lists the same pattern in four files, about 24 instances in total, with admin.py holding around 20 of them.

The fix

PR #1886 nests the commit in its own try so the close survives a failing commit:

finally:
    try:
        db.commit()
    finally:
        db.close()   # always runs

Under the same 3,000-user load:

	Before the fix	After the fix
Stuck connections	42 to 81	0
Connection release	Held up to 120 s	Cycling in 1 to 2 s
Pool state	Exhausted	Healthy

The rule underneath: if you acquire a finite production resource, its release has to survive every failure path.

The experiment: could the code have told us first?

Everything the team learned came from production-scale conditions. The question was whether the weakness was visible in the code alone, before any load test.

We checked out commit cf5adcd5d, the code exactly as it stood before the fix, and gave Tomosu one file: token_scoping.py, 755 lines. Tomosu did not get the issue, the PR, the load-test results, the release notes, or any hint of what to look for.

Before scanning, we sealed the success criterion so the result couldn't be reinterpreted afterwards:

Identify unsafe database-session cleanup where an operation before close() can fail and prevent reliable connection release.
Direct hit: names the unsafe session lifecycle on this path. Adjacent: generic exception-handling or database advice. Miss: anything else.

The result

One file, 1 minute 41 seconds, 5 findings, a Production Reliability Index of 27 out of 100. One of the five, rated high severity:

Unsafe database-session lifecycle. An operation performed before the session is closed can fail, leaving the connection unreleased.

It names the cleanup path, says the preceding operation can fail, and ties that to connection release. All three checks in the criterion pass.

Tomosu's suggested change went a different way from IBM's. This path only reads (the stuck query was a SELECT), so it drops the commit entirely:

finally:
    db.close()

Three versions of the same block, one invariant:

Before: release depends on the commit succeeding.
IBM's fix: commit kept, close protected by its own finally.
Tomosu, blind: commit removed, so there is nothing for release to depend on.

What this does and doesn't show

It shows the weakness was readable from the source, with no incident context, before anyone ran a load test.

It doesn't show Tomosu would have found all ~24 instances: we scanned one file, and we chose that file knowing where the bug was. A full-repository blind scan at the same commit is the stronger test. Static analysis also doesn't replace load testing. It moves some of what load testing teaches earlier, to the pull request, where the fix costs three lines instead of an outage.

If you use SQLAlchemy sessions by hand, search your code for commit() followed by close() in the same finally. A context manager (with SessionLocal() as db:) or a nested try/finally closes the hole.

Tomosu scores each pull request with a 0 to 100 Production Reliability Index before it merges. Run a free scan on one public repository, or read the full case study with the 22-slide forensics deck.

ContextForge is an open-source project. This analysis uses its public repository history; Tomosu AI is not affiliated with IBM. Originally published on tomosu.ai.

What AI Reliability Actually Means (and How to Measure It)

Parsa Mohammadi — Tue, 13 Oct 2026 13:50:09 +0000

AI reliability is how consistently an AI system, or software an AI has changed, does the correct thing under real production conditions, measured over time. Most writing on the term covers only the first half: whether a model gives good answers. For engineering teams in 2026 the second half matters as much, because agents now write and merge a growing share of the code that runs in production.

Short version: reliability for AI has two halves.

Runtime half: whether a model, an LLM app or an agent behaves correctly and consistently while it runs. Judged with evals, tracing, guardrails and consistency across runs.
Change half: whether the code, config and infrastructure an agent produces behave correctly once merged and deployed. Judged with review, tests, merge gates and change risk scoring.
Where they meet: an agent that acts, because its output is a change.

Uptime measures neither. You need a quality signal next to availability, and a change-level view of what agents ship.

NIST's AI Risk Management Framework borrows the ISO/IEC definition of reliability: the ability of an item to perform as required, without failure, for a given time interval, under given conditions (NIST AI RMF). That definition fits both halves. What changes is which "item" you're judging.

The two meanings of AI reliability

	Reliability of AI systems	Reliability of AI-written changes
What is being judged	A model, LLM app or agent at runtime	Code, config or infra an agent produced
Typical failure	Wrong, inconsistent or unsafe output	A merged change that breaks production
Who notices first	Users, evals, support tickets	On-call, error rates, rollbacks
Main tools	Evals, tracing, guardrails	Review, tests, merge gates, change risk scoring

 AI SYSTEM AT RUNTIME            AI-WRITTEN CHANGE
 model, LLM app, agent           code, config, infrastructure
 judged by evals,                judged by review, tests,
 tracing, guardrails             merge gates
            \                        /
             \                      /
          AN AGENT THAT ACTS: its output is a change
      (both halves apply when an agent edits, migrates or deploys)

The two meet when an agent takes actions. An agent that edits files, runs migrations or calls production APIs is an AI system whose output is a change. For the broader picture on the second column, see What Is Production Reliability in Software Engineering?

Is uptime the same as reliability for an LLM?

No. A traditional service is mostly reliable when it's up and fast. An LLM endpoint can return 200 OK in 400 ms with a wrong answer, and the availability dashboard stays green the whole time.

Uptime is necessary, not sufficient. An LLM feature needs at least one quality signal next to the infrastructure ones. Without it, a prompt change that quietly lowers answer quality looks like a perfect week.

How to measure AI reliability

Google's SRE book defines an SLI as a carefully defined quantitative measure of some aspect of a service, and an SLO as a target for it (SRE book). The same approach works for AI, with a few extra indicators.

SLI	What it catches	How to compute it
Availability	Provider or gateway outages	Successful responses / total requests
Latency (p95)	Slow tool loops, long generations	p95 of end-to-end request time, including retries
Eval pass rate	Quality regressions after a prompt or model change	Pass rate on a fixed, versioned eval set, run on every change
Consistency	Answers that flip between runs	Agreement rate across N runs of the same input
Guarded-action rate (agents)	Agents attempting risky actions	Tool calls blocked by policy / total tool calls
Change fail rate (AI-authored)	Agent PRs that break production	DORA's change fail rate, filtered to AI-authored changes

DORA defines change fail rate as the ratio of deployments that need immediate intervention after they go out (DORA). Splitting it by author type is the simplest way to see whether agent-written changes are more or less reliable than human ones in your codebase. A PR label or the bot's author name is usually enough. How to Measure Software Change Reliability goes deeper on the change-level metrics.

Why agents turn reliability into a change problem

In July 2025, SaaStr founder Jason Lemkin described Replit's agent deleting his production database during a code freeze, then telling him a rollback wouldn't work when it actually did (The Register). Model quality wasn't the issue. The failure was an agent with write access to production and nothing between its decision and the database.

That's the part of AI reliability most teams under-invest in. Evals tell you whether the model is good on average. They don't tell you whether this specific change, to this specific service, is safe to ship. That question belongs to change risk management.

How to set reliability targets for AI features

Pick one quality SLI per feature, not five. Eval pass rate is usually the right first one.
Version the eval set and run it in CI on every prompt, model or retrieval change.
Set the SLO from a baseline week of data, not a guess.
Tag AI-authored PRs so change fail rate can be split by author.
Decide which failures block a merge or a deploy, and which only alert.

Common mistakes

Treating a provider's status page as your reliability number.
Running evals once at launch and never again.
Measuring the model while ignoring the code the agent ships.
Averaging consistency across all inputs, which hides the few prompts that flip every time.

Tomosu works on the change half. It scores each pull request with a 0 to 100 Production Reliability Index before it merges, with an advisory Ready or Not Ready to Merge verdict at 70. A plain scan covers the code; the production-side sub-indices need observability and ticketing integrations connected. Run a free PRI scan on one repository.

Originally published on tomosu.ai.

Connection Leak vs. Pool Exhaustion: How to Tell the Difference

Parsa Mohammadi — Mon, 12 Oct 2026 13:48:31 +0000

Every connection is checked out, requests are queuing, and the pool is throwing timeouts. The next decision depends on one question: is something leaking connections, or is the pool being asked for more than it can give? Both produce the same error. They need opposite fixes.

Short version: pool exhaustion is a state (every pooled connection is checked out, new callers wait). A connection leak is one cause of it (code checks out a connection and never returns it). Watch in-use connections against traffic. A leak ratchets up and stays up. Load-driven exhaustion falls when traffic falls.

Leak: in-use climbs with uptime, ignores quiet periods, resets on restart.
Load or slow work: in-use tracks request rate and query time, and recovers.
Confirm at the database: leaked connections usually sit idle while the pool counts them as busy.
Fix a leak in code (release on every path), not with a bigger pool.

The mechanism is the same for HikariCP on the JVM, pg.Pool in Node.js, SQLAlchemy's QueuePool in Python, and Go's database/sql. Where the tools differ, there's a per-stack table below. For a JVM-only walkthrough of HikariCP's exception, leak detection and thread dumps, see HikariCP "Connection Is Not Available, Request Timed Out".

Leak vs. exhaustion: the definitions

Pool exhaustion is the condition where every connection a pool may hold is checked out, so the next caller waits and eventually times out. It's a symptom.

A connection leak is a specific bug: a code path checks out a connection and never returns it, so usable capacity shrinks by one each time that path runs. A leak always ends in exhaustion if the process lives long enough. Exhaustion does not imply a leak.

The other causes of exhaustion return every connection eventually, just not fast enough:

Load: request rate × normal hold time exceeds pool size at peak.
Slow queries: a missing index, a lock wait, or a bigger table stretches each hold.
Long holds in application code: a transaction that spans an HTTP call, a queue publish, or slow processing.
Nested acquisition: a thread holding one connection waits for a second, which can deadlock the pool at modest concurrency.
Amplified demand: client retries add requests exactly when each request is slowest.

Search Stack Overflow for pool exhaustion and most threads end with "increase the pool size" or "you have a leak", usually without evidence for either. The useful question is "leak, or one of the causes that recover on their own?" A leak needs a code change on a specific path. The others need faster work, shorter holds, less demand, or (last) more connections.

Property	Connection leak	Exhaustion without a leak
Every connection eventually returned?	No. Stays checked out until the process exits	Yes, after the query, transaction, or remote call finishes
In-use when traffic drops	Stays at its new high	Falls back to a low baseline
Correlates with	Uptime, and the rate of one code path (often an error path)	Request rate, query latency, dependency latency
Effect of a restart	Fixes it completely, until the leak catches up	Little or none if the load or slowness persists
Effect of a bigger pool	Delays the outage	Helps only if the pool is truly undersized

What they look like over a day

At the moment of the timeout both look identical: in-use equals max, idle is zero, callers are waiting. The difference only shows up as a time series. Put in-use connections and request rate on the same chart over at least one daily cycle.

CONNECTION LEAK (pool max = 10)
in-use
 10 |              ┌─────────────┐
    |         ┌────┘   traffic   │      steps up, never steps down,
  5 |    ┌────┘      falls, but  │      even while traffic falls
    |┌───┘           in-use stays│  ┌───
  0 +────────────────────────────┴──┘───   ← restart resets it
     00:00   06:00   12:00   18:00   24:00

EXHAUSTION FROM LOAD (no leak, pool max = 10)
in-use
 10 |              ┌───┐   ← pool full at peak, callers wait
    |           ┌──┘   └──┐
  5 |        ┌──┘         └──┐
    |    ┌───┘               └───┐
  0 +────┘                       └──────   recovers as traffic falls
     00:00   06:00   12:00   18:00   24:00

Three details carry most of the diagnosis:

Does in-use come down in the quiet hours? If traffic drops 80% overnight and in-use doesn't move, those connections aren't doing work for anyone.
What does a restart do? Instant drop to baseline followed by a slow climb is the classic leak signature. If the pool is full again within minutes of a restart, look at load or slow work first.
Where are the steps? Leaks step up when the leaking path runs. If steps line up with a batch job, a retry burst, or error spikes on one endpoint, you have a lead on which path it is.

Autoscaling and deploys hide leaks. If instances get replaced often, a slow leak may never reach the pool limit. It shows up when deploys pause over a holiday, or when a failing dependency makes the leaking path run far more often. Chart in-use per instance against instance uptime, not only the fleet total, or the leak hides inside the average.

Exhaustion in Java, Node.js, Python, and Go

Every pool has a max size, a way to wait for a free connection, and some way to report how many are checked out. The defaults differ in ways that change what the incident looks like. Two of the four common stacks don't fail fast by default.

Stack	Pool size (default)	Wait limit (default)
JDBC · HikariCP	`maximumPoolSize` (10)	`connectionTimeout` (30 s)
Node.js · `pg.Pool`	`max` (10)	`connectionTimeoutMillis` (0 = wait forever)
Python · SQLAlchemy `QueuePool`	`pool_size` (5) + `max_overflow` (10)	`pool_timeout` (30 s)
Go · `database/sql`	`SetMaxOpenConns` (0 = unlimited)	None, bounded only by the caller's context

Stack	What exhaustion looks like	Where to read in-use
JDBC · HikariCP	`SQLTransientConnectionException`: "Connection is not available, request timed out"	Pool stats in the message, `hikaricp.connections.active`, `leakDetectionThreshold` for borrower traces
Node.js · `pg.Pool`	Requests hang. With a timeout set: "timeout exceeded when trying to connect"	`pool.totalCount`, `pool.idleCount`, `pool.waitingCount`
Python · SQLAlchemy	`TimeoutError`: "QueuePool limit of size 5 overflow 10 reached, connection timed out"	`engine.pool.status()`, `checkedout()`, `checkout`/`checkin` pool events
Go · `database/sql`	Goroutines block, then `context deadline exceeded`. If unlimited, the database refuses new connections instead	`db.Stats()`: `InUse`, `Idle`, `WaitCount`, `WaitDuration`

Defaults as documented by HikariCP, node-postgres, SQLAlchemy, and Go. Frameworks often override them, so read the effective values from your running config.

node-postgres waits forever by default

With connectionTimeoutMillis at 0, a caller that can't get a client waits with no limit. An exhausted pool doesn't throw. Requests just stop completing, which usually surfaces as a load balancer or gateway timeout far from the real cause. Set a timeout so the failure is visible, and export the counters the pool already has:

const pool = new Pool({
  max: 10,
  connectionTimeoutMillis: 5_000,   // default 0 means wait forever
  idleTimeoutMillis: 10_000,
});

setInterval(() => {
  metrics.gauge('db.pool.total', pool.totalCount);
  metrics.gauge('db.pool.idle', pool.idleCount);
  metrics.gauge('db.pool.waiting', pool.waitingCount);
  metrics.gauge('db.pool.in_use', pool.totalCount - pool.idleCount);
}, 10_000).unref();

Go's database/sql is unlimited by default

With SetMaxOpenConns at 0, a Go service never exhausts its own pool. A leak opens new connections until the database refuses them (PostgreSQL's "sorry, too many clients already", MySQL's "Too many connections"). That error can hit every other service sharing the database, so one service's leak looks like a platform-wide outage. Set a limit and log the stats:

db.SetMaxOpenConns(20)             // default 0 = unlimited
db.SetMaxIdleConns(20)             // default 2
db.SetConnMaxLifetime(30 * time.Minute)

go func() {
    t := time.NewTicker(15 * time.Second)
    defer t.Stop()
    for range t.C {
        s := db.Stats()
        log.Printf("db pool open=%d inuse=%d idle=%d waits=%d waited=%s",
            s.OpenConnections, s.InUse, s.Idle, s.WaitCount, s.WaitDuration)
    }
}()

On any stack the four numbers that matter are max, in use, idle, and waiting (or cumulative wait count and time). If you add one metric after reading this, make it in-use as a time series.

Confirm a leak from the database side

The pool knows which connections are checked out. The database knows what each of them is actually doing. Comparing the two is the fastest confirmation, and it works on any stack.

SELECT application_name, state, count(*) AS sessions,
       max(now() - state_change) AS longest_in_state
FROM   pg_stat_activity
WHERE  datname = current_database()
  AND  backend_type = 'client backend'
GROUP BY 1, 2
ORDER BY 3 DESC;

Column meanings are in the PostgreSQL cumulative statistics docs. Set application_name in each service's connection string so you can filter to one service's pool, then line the result up against the pool's counters at the same moment.

Example with a pool of 10 (illustrative numbers):

View	What it reports
Pool	10 in use, 0 idle
Database	2 `active`, 1 `idle in transaction`, 7 `idle`

The 2 active sessions are running queries right now. The 1 idle-in-transaction session has an open transaction and no query, so check its age. The 7 idle sessions the pool counts as busy were borrowed and never given back. Those are your leak candidates.

Pool says	Database says	Most likely cause
All in use	Most sessions `idle`, `state_change` minutes or hours old	Leak with autocommit: connections borrowed and abandoned
All in use	Many `idle in transaction`, old `xact_start`	Leak mid-transaction, or a long hold around non-database work. Age tells them apart
All in use	Most sessions `active`, long `query_start` age	Slow queries or a lock chain. Check `wait_event_type`
All in use	Most sessions `active`, queries short	Load: throughput × hold time exceeds the pool
Below max, callers waiting	Fewer sessions than expected, connection errors logged	Creation failure, e.g. the database's connection limit reached by other clients

MySQL shows the same picture in SHOW FULL PROCESSLIST (Sleep with a large Time) and information_schema.innodb_trx.

A database timeout protects the database, not the pool. PostgreSQL's idle_in_transaction_session_timeout ends sessions that sit in an open transaction too long and releases their locks. Good guardrail. It does not return the connection to your pool: the pool still counts it as checked out until application code releases it, and the next use of that connection fails. Damage control, not a leak fix.

Why leaks show up during a different incident

Most real leaks aren't on the happy path. The happy path runs in every test and every request, so a missing release there gets found within hours. Leaks survive on error and early-return paths: a row fails to parse, a downstream call throws, validation fails, a request is cancelled. So a leak drains the pool at the rate its path runs.

Time to drain = pool size ÷ leak rate (illustrative numbers):

Situation	Error path rate	Pool of 20 drains in
Normal day	~2 per hour	~10 hours (daily deploys restart it first, so nobody notices)
Dependency failing	~5 per second	~4 seconds

This is why "the pool ran out during the payment provider outage" is often a misdiagnosis. The outage was real. The exhaustion was a second, pre-existing bug the outage exposed. The tell is what happens after the dependency recovers: exhaustion from load or slowness clears within seconds, while a leak leaves the pool full until someone restarts the service.

The same logic covers leaks that appear "right after a deploy". The deploy may not have added the leak. It may have added a new failure mode, stricter validation, or a new timeout that sends more traffic down an old error branch. When you're hunting for the commit behind an incident, include changes that increase the rate of existing error paths, not only changes to data access code.

Code patterns that leak connections

Same across languages: acquisition without a guaranteed release, plus resources that pin a connection (result sets, cursors, streams, transactions) not closed on every path.

BROKEN
acquire() ──► query + map rows ──► release() ──► return
                    │
                    └── throws ──► error propagates, release() skipped
                                   (connection stays checked out forever)

FIXED
acquire() ──► try: query + map ──► finally: release() ──► exit
                    │                     ▲
                    └── throws ───────────┘
              (success and error both pass through finally)

The leak is a missing edge in the control flow. Structured release (try/finally, try-with-resources, with, defer) adds it for every path at once.

Node.js: `client.release()` skipped when a query throws

Leaks when a query throws:

async function transfer(from, to, amount) {
  const client = await pool.connect();
  await client.query('BEGIN');
  await client.query(DEBIT_SQL, [from, amount]);   // may throw
  await client.query(CREDIT_SQL, [to, amount]);
  await client.query('COMMIT');
  client.release();                                  // never reached on error
}

Released on every path:

async function transfer(from, to, amount) {
  const client = await pool.connect();
  let broken;
  try {
    await client.query('BEGIN');
    await client.query(DEBIT_SQL, [from, amount]);
    await client.query(CREDIT_SQL, [to, amount]);
    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK').catch((e) => { broken = e; });
    throw err;
  } finally {
    client.release(broken);   // always runs; truthy arg destroys it
  }
}

For single statements, pool.query() checks out and releases a client for you, so it can't leak. The risk lives in code that calls pool.connect() directly, usually for transactions.

Go: `rows` or `tx` left open on an early return

Leaks on a Scan error:

rows, err := db.QueryContext(ctx, listItemsSQL, orderID)
if err != nil {
    return nil, err
}
for rows.Next() {
    var it Item
    if err := rows.Scan(&it.ID, &it.SKU, &it.Qty); err != nil {
        return nil, err   // rows still open: its connection stays in use
    }
    items = append(items, it)
}
return items, rows.Err()

Closed on every path:

rows, err := db.QueryContext(ctx, listItemsSQL, orderID)
if err != nil {
    return nil, err
}
defer rows.Close()   // safe to call even after Next has returned false

tx, err := db.BeginTx(ctx, nil)
if err != nil {
    return err
}
defer tx.Rollback()  // no-op after a successful Commit

When rows.Next() returns false the rows close automatically, which is why the happy path in the broken example doesn't leak. Only the early return does, and only when a Scan fails. Exactly the kind of path tests rarely hit.

Python: a Session not closed on an exception

# Broken: close() is skipped if anything between here and there raises
def load_invoice(invoice_id):
    session = SessionLocal()
    invoice = session.get(Invoice, invoice_id)
    render_pdf(invoice)                 # raises: session keeps its connection
    session.close()

# Fixed: the context manager closes the Session on every path
def load_invoice(invoice_id):
    with SessionLocal() as session:
        invoice = session.get(Invoice, invoice_id)
        render_pdf(invoice)             # connection returned when the block exits

In CPython, reference counting sometimes returns that connection when the abandoned object is garbage collected, and SQLAlchemy may log a warning when it cleans up a connection that was never checked in. That makes Python leaks intermittent rather than absent. Don't rely on it. FastAPI dependencies with yield, or scoped_session.remove() at request end, exist to make release structural.

Other patterns that pin a connection

Streams and cursors returned to the caller: a repository method that returns a lazy stream or server-side cursor keeps its connection until the caller closes it.
Transactions left open by an early return between BEGIN and COMMIT, which also hold locks.
Connections stored in fields or passed to another thread, where ownership of the release is unclear.
Background jobs that open a connection outside the request lifecycle, so request-scoped cleanup never runs.

Track the borrower when your pool won't

HikariCP has leakDetectionThreshold. node-postgres has no built-in equivalent, but you can add one in a few lines by recording where each client was checked out:

const held = new Map();

async function connect() {
  const client = await pool.connect();
  held.set(client, { at: Date.now(), stack: new Error('checked out here').stack });
  const release = client.release;
  client.release = (err) => { held.delete(client); return release(err); };
  return client;
}

setInterval(() => {
  const now = Date.now();
  for (const { at, stack } of held.values()) {
    if (now - at > 30_000) console.warn(`db client held ${now - at} ms`, stack);
  }
}, 10_000).unref();

A warning means "held a long time", not "leaked". A warning for a client that's later released is a long hold. A warning that repeats from the same stack and never clears is a leak. In SQLAlchemy, the pool's checkout and checkin events give you the same hook.

Step by step: leak or exhaustion?

Read the error and the pool counters. Record max, in use, idle, and waiting at the moment of failure, plus any chained cause. If total is below max while callers wait, the pool is failing to create connections, which is a separate problem.
Chart in-use against request rate over a full day, per instance, against instance uptime. A ratchet that ignores quiet periods and resets on restart points to a leak. A curve that follows traffic points to load or slow work.
Compare the pool's view with the database's. Many connections checked out in the pool but idle or long idle in transaction at the database confirms a leak. Mostly active sessions point to load or slow queries.
Turn on borrower tracking. Use the pool's leak detection, or a wrapper that records checkout stack and age. Keep the warnings that never clear; they name the leaking path.
Find every acquisition path without structured release. Search for manual checkouts, transactions, returned cursors, and streams. Check error and early-return branches, not just the happy path.
Fix, then verify the shape. After the fix ships, in-use should return to baseline in quiet periods and after the next dependency failure. If it still ratchets, there's a second path.

Fixes, matched to the cause

Cause	Fix	What not to do
Connection leak	Release structurally on every path (try/finally, try-with-resources, `with`, `defer`). Close returned cursors and streams. Prefer single-call APIs like `pool.query()`	Raise the pool size or restart on a schedule. Both hide the leak until the error rate rises
Leak mid-transaction	Roll back on every non-commit path. Add `idle_in_transaction_session_timeout` as a lock guardrail	Rely on the database timeout as the fix; the pool still thinks the connection is out
Slow queries	Fix the query plan or index; add a statement timeout so one query can't hold a connection indefinitely	Raise the pool wait timeout. Users wait longer for the same failure
Long holds around remote calls	Move I/O out of the transaction and add client timeouts to the remote call	Add connections to match the slowest dependency
Load or amplified demand	Reduce demand (cache, batch, bound retries), shed load, then size the pool from throughput × hold time within the database's limit	Size each instance's pool without multiplying by instance count
Undefined limits	Set a pool max and a wait timeout explicitly (Go `SetMaxOpenConns`, node-postgres `connectionTimeoutMillis`)	Leave defaults that wait forever or open unlimited connections

Treat any change to pool size, pool timeout, or data access code as a change to a shared production resource. Several services and background jobs usually share the database's connection budget, so a local "fix" can move the exhaustion somewhere else.

Key takeaways

Pool exhaustion is a symptom. A leak is one cause; load, slow queries, long holds, and nested acquisition are others.
Chart in-use against request rate, per instance. A leak ratchets and ignores quiet periods. Load-driven exhaustion recovers.
Connections checked out in the pool but idle at the database are leak candidates.
Most leaks live on error and early-return paths, so they drain the pool at the error rate and surface during other incidents.
node-postgres waits forever and Go opens unlimited connections by default. Set explicit limits so exhaustion is visible.
Fix leaks with structured release on every path. A bigger pool only delays a leak.

Runtime metrics tell you a pool is leaking, but a path only shows up in a leak trace after it has run in production. Tomosu checks the repository and each pull request for that kind of risk before merge: acquisition paths where an error branch can skip the release, holds that stretch across remote calls, and which endpoints share the pool. A PR that adds a pool.connect() without a finally shows up in its Production Reliability Index at review time instead of during the next dependency outage. Scan a repository with Tomosu.

Originally published on tomosu.ai, part of the Production Debugging series.

Change Risk Management in Software Engineering

Parsa Mohammadi — Thu, 08 Oct 2026 20:00:00 +0000

Software changes are unavoidable.

The goal of change risk management is not to stop changes.

It is to understand the potential impact of a change and apply the right controls before and during deployment.

What is change risk management?

Change risk management is the process of identifying, evaluating, and managing the potential consequences of a software change.

That can include:

Code changes
Configuration changes
Database changes
Infrastructure changes
Dependency upgrades
API changes
Deployment changes

Start before deployment

The earlier you identify uncertainty, the more options you have.

Before merge or deployment, look at:

Change scope
Dependencies
Testing
Production usage
Incident history
Runtime signals
Blast radius
Rollback conditions

Use evidence rather than intuition

A useful assessment should answer:

What do we know?

What don't we know?

What could this change affect?

What controls are available?

This creates a more explicit basis for the deployment decision.

Match controls to the change

Not every change needs the same process.

Possible controls include:

Additional code review
More integration testing
Staged deployment
Feature flags
Increased monitoring
Rollback preparation
Additional approval

The appropriate control depends on the change and its context.

Production reliability and change management

Traditional change management often focuses on approvals and process.

Production reliability adds technical evidence around the change itself.

That includes how the code relates to dependencies, production behavior, historical incidents, and potential blast radius.

AI assisted development changes the volume

AI coding tools can increase the number of changes engineers can produce.

That can make change risk management more dependent on automated signals and prioritization.

The goal is not to slow development down.

It is to make it easier to identify which changes need more attention.

A practical framework

Change
  ↓
Production context
  ↓
Evidence
  ↓
Potential impact
  ↓
Controls
  ↓
Deployment
  ↓
Production feedback

The last step matters because deployment creates new evidence that can inform future changes.

Where PRI fits

Tomosu's Production Reliability Index (PRI) provides a summary signal around software changes using multiple reliability indicators.

It can be used as one input into a broader change management process.

Run a PRI assessment:

https://tomosu.ai/start

The main idea

Good change risk management is not about making every change look risky.

It is about distinguishing changes that can move through the normal workflow from changes where additional evidence or controls are useful.

The better the context, the better the decision.

How to Prioritize Risky Pull Requests

Parsa Mohammadi — Wed, 07 Oct 2026 20:00:00 +0000

When a team has dozens of pull requests, not every change can receive the same amount of attention.

The useful question is:

Where will additional review, testing, or investigation provide the most value?

Prioritization should start with potential production impact, not simply the number of changed lines.

Start with production impact

Ask what the change could affect.

A change to a shared payment component deserves different attention from a change to an isolated internal tool.

Look at change scope

Understand which files, services, APIs, databases, and infrastructure are involved.

Then look for indirect effects.

Check dependencies

A small change can affect many services when it sits low in a dependency graph.

Dependency reach is therefore an important prioritization signal.

Check production usage

How heavily is the affected component used?

How important are the workflows that depend on it?

Check testing evidence

Limited testing increases uncertainty.

Strong evidence can reduce the amount of additional investigation required.

Check incident history

Previous incidents and rollbacks provide context about areas that may deserve closer attention.

Look at runtime behavior

Current production signals can help identify changes touching areas that are already behaving abnormally.

Consider recent activity

Frequent changes, recent regressions, or rapidly evolving components can add context.

Assess blast radius

Think about the systems, users, workflows, and dependencies that could be affected.

Consider deployment conditions

A change that can be safely rolled back or gradually deployed may be easier to manage than one with the same potential impact but no practical rollback path.

A practical prioritization model

Potential impact
      +
Evidence
      +
Uncertainty
      +
Deployment control
      ↓
Attention required

This is a prioritization model, not a universal formula.

Risky does not automatically mean blocked

A change can deserve additional attention without being rejected.

Possible actions include:

Additional review
More integration testing
Staged deployment
Feature flags
Additional monitoring
Rollback preparation

Where PRI fits

The Production Reliability Index (PRI) can summarize multiple signals so teams can focus investigation where the evidence suggests more attention may be useful.

Run a PRI assessment:

https://tomosu.ai/start

How to Identify High Risk Pull Requests

Parsa Mohammadi — Tue, 06 Oct 2026 20:00:00 +0000

Not every pull request deserves the same level of scrutiny.

A useful review process should help engineers find the changes where additional investigation is most valuable.

A high risk pull request is not simply a large pull request.

It is a change with characteristics that suggest greater potential production impact or uncertainty.

Start with change scope

Look at what the PR actually changes.

Pay attention to:

Shared components
Critical services
Databases
Authentication
Infrastructure
Customer facing behavior
APIs

Look at dependencies

A change can become more important when many systems depend on the affected component.

Dependency reach is often more informative than diff size.

Check production usage

Ask how the changed code is used in production.

A rarely used internal function and a heavily used payment path should not be treated as equivalent.

Check testing

Look at the quality and coverage of the evidence.

A change with limited testing around an important production path may deserve more attention.

Look at recent activity

Frequent changes can indicate an area that is evolving quickly.

Combine this with incident history and current change scope rather than using churn alone as a risk rule.

Check runtime signals

Production behavior can provide additional context.

Look for unusual errors, latency changes, dependency failures, or other anomalies in the affected area.

Check incident history

Previous incidents and rollbacks can help identify areas where a change deserves closer investigation.

Assess blast radius

Ask:

If this change behaves unexpectedly, what could it affect?

Think about services, users, workflows, dependencies, and business functions.

Consider rollback

A change that can be quickly rolled back is operationally different from one that is difficult to reverse.

Large PR does not automatically mean high risk

Diff size is only one signal.

A large isolated change may have limited impact.

A tiny change to a shared critical component may have much greater consequences.

Where PRI fits

Tomosu's Production Reliability Index (PRI) combines multiple signals around a change to help engineers identify where additional attention may be useful.

Run a PRI assessment:

https://tomosu.ai/start

How to Assess the Blast Radius of a Code Change

Parsa Mohammadi — Mon, 05 Oct 2026 20:00:00 +0000

A code change can be small in lines of code and large in production impact.

That is the basic idea behind blast radius.

The blast radius of a change is the set of systems, components, users, or business functions that could be affected if the change behaves unexpectedly.

Start with the changed component

Identify exactly what the pull request modifies.

Then ask:

Is it shared?
Is it customer facing?
Is it part of a critical workflow?
Does it interact with a database?
Does it handle authentication?
Does it control infrastructure?

Map direct dependencies

Find the components that directly depend on the changed code.

Then look one level further.

Changed component
      ↓
Direct dependencies
      ↓
Dependent services
      ↓
Customer workflows

The goal is to understand propagation.

Look at indirect impact

Some changes affect components that don't appear in the pull request.

For example, changing a shared API contract can affect multiple services even when those services are untouched.

That is why dependency mapping matters.

Check production usage

Not every dependency has equal importance.

Ask:

How heavily is it used?
Which customers depend on it?
Which workflows rely on it?
Is it part of a critical path?

A shared component with low production usage may have a smaller practical blast radius than a heavily used component.

Look at history

Previous incidents can reveal where changes have caused problems before.

Also look at recent changes and regressions in the same area.

Consider the deployment

Blast radius also depends on how a change is deployed.

A gradual rollout, feature flag, or quick rollback can limit the impact of a failure.

Diff size is not blast radius

A useful rule is:

Lines changed ≠ production impact

A five line change to a shared authentication library can matter more than a 500 line change to an isolated internal tool.

Where PRI fits

The Production Reliability Index (PRI) can use change context, dependencies, production signals, and other reliability evidence to help surface changes that deserve additional attention.

Run a PRI assessment:

https://tomosu.ai/start

Hacktoberfest tip before you submit: break one function on purpose (return []) and run your tests. If the suite still passes, those tests check nothing. Fix them or delete them, a fake green is worse than no test. #hacktoberfest

Parsa Mohammadi — Mon, 05 Oct 2026 12:17:44 +0000

Agent wrote the code and the tests? Break one function on purpose. If tests pass, they're fake.

Parsa Mohammadi — Mon, 05 Oct 2026 10:23:06 +0000

Hacktoberfest: Contribution Chronicles

Parsa Mohammadi

Oct 5

Your Hacktoberfest tests are green. That doesn't mean they ran.

#discuss #hacktoberfest #ai #testing

3 min read

Your Hacktoberfest tests are green. That doesn't mean they ran.

Parsa Mohammadi — Mon, 05 Oct 2026 09:35:59 +0000

TL;DR: If an agent wrote your code and your tests, a green build means the agent is satisfied. Whether the project works is a separate question. Break one function on purpose before you submit. If the suite still passes, those tests aren't checking anything. Checklist and a free scan at the end.

More and more Hacktoberfest projects are built with coding agents. That's fine. The risky moment is the last hour, when the agent says "all tests passing ✅" and you believe it because you're tired and the deadline is tonight.

Coding agents are generally optimized to satisfy the checks they're given. If the tests are weak, an agent can produce a green build without proving much about the implementation. Sometimes it gets there by making the code work. Sometimes it gets there by making the tests stop checking anything.

What a fake green looks like

None of these fail. All of them look fine at a glance.

The assertion that can't fail:

def test_parse_transactions():
    result = parse_transactions(SAMPLE_SMS)
    assert result is not None  # it never returns None, so this passes forever

The test that quietly left:

@pytest.mark.skip(reason="flaky, fix later")
def test_handles_empty_input():
    ...

The mock that answers its own question:

@patch("app.model.classify", return_value="groceries")
def test_classifies_groceries(mock_classify):
    assert categorize("grocery order #4412") == "groceries"
    # the real model never ran

The try/except that eats the failure:

def test_model_loads():
    try:
        load_model("gemma-3-4b-q4.gguf")
    except Exception:
        pass

The one check that catches most of it

Break your own code on purpose:

def parse_transactions(text):
    return []  # temporary sabotage

Run the suite. If it still passes, those tests weren't testing that function. Revert, then fix the tests or delete them. A deleted fake test is more honest than a green one.

The rest of the pre-submit list

Put agent work through pull requests, even solo. You get a diff to read, and your commit history shows the project was built inside the challenge window, which is a rule.
Search for secrets before the repo goes public: sk-, api_key, token, and every service name you used. If a key was ever committed, rotate it. Deleting the line doesn't remove it from git history.
Clone fresh and follow your own README. Judges will. The usual misses are a model file nothing downloads, an undocumented env var, and a globally installed dependency.
Write down the model: name, size, quantization, and the hardware you tested on. The prompt asks why open-source AI matters for your project, and "runs on a laptop" is a claim readers will check.
Note post-deadline commits in your README. The challenge FAQ says skipping this can get an entry disqualified.
Credit borrowed code. If your agent pulled a chunk of someone else's project in, find it and credit it.

A second check before you submit, free

If you'd rather not do all of this by eye at 11pm, Tomosu AI can scan the project for you. There are two free ways in.

Small repo (under 50 files): scan it online. Paste your GitHub repo URL at tomosu.ai/start. Public repos scan without a login, and the scan is read-only. Most one-week hackathon projects fit under the limit.

Bigger repo: use the free editor extension. Install it in VS Code, Cursor, or Antigravity and scan your workspace from the editor:

VS Code Marketplace
Open VSX (for Cursor, Antigravity, and other VS Code forks)
Installation guide

Either way you get a Production Reliability Index (PRI) score for the project. Run it once before your final commit, fix what it flags, and run it again.

A screenshot of the first scan, the second scan, and what you changed in between makes a good "show your work" section. Judges weight the write-up most heavily, and a real before/after tells them more about your process than a feature list.

Over to you

What's the sneakiest thing an agent has done to get your tests green? Drop it in the comments. The best ones go into a follow-up post, with credit.

Pre Merge Reliability Analysis

Parsa Mohammadi — Sun, 04 Oct 2026 20:00:00 +0000

Most code review happens before merge.

That makes the pull request a useful place to assess more than implementation quality.

A pre merge reliability analysis looks at the proposed change together with the production context around it.

The goal is simple:

Understand the potential production impact before the change becomes part of the next release.

Start with change scope

Identify what the pull request changes.

Look at:

Files
Services
APIs
Databases
Infrastructure
Shared components
Customer facing behavior

Then ask what is indirectly affected.

Map dependencies

A change can propagate through dependencies.

PR
 ↓
Changed component
 ↓
Dependencies
 ↓
Production services
 ↓
Users and workflows

This is often more useful than looking at the number of changed lines.

Review testing evidence

Ask whether the important behavior is actually verified.

Consider:

Unit tests
Integration tests
End to end tests
Regression tests
Failure scenarios
Dependency behavior

A green test suite is evidence.

It is not proof that every production condition has been covered.

Add production context

Look at how the affected components behave in production.

Consider traffic, customer usage, runtime anomalies, and operational constraints.

Check history

Previous incidents, rollbacks, and repeated regressions can provide useful context.

The goal is not to automatically reject changes in areas with a history of problems.

The goal is to make that history visible during review.

Consider deployment conditions

Ask whether the change can be safely deployed and reversed.

A change with a large potential impact may have manageable operational risk if it can be deployed gradually and rolled back quickly.

What should block a merge?

There is no universal threshold.

Different teams have different systems, deployment processes, and tolerance for uncertainty.

The useful question is:

Does the available evidence justify merging this change under the current deployment conditions?

Where PRI fits

The Production Reliability Index (PRI) can provide a summary signal around the change while preserving the underlying findings that explain it.

That makes it useful as another input into the merge decision rather than a replacement for review.

Run a PRI assessment:

https://tomosu.ai/start

How to Measure Software Change Reliability

Parsa Mohammadi — Thu, 01 Oct 2026 20:00:00 +0000

Teams have plenty of metrics for services.

They measure availability, latency, error rates, deployment frequency, and incident recovery.

But there is another useful question:

How reliable is this particular software change?

That is a change level question rather than a service level question.

Service reliability vs change reliability

Service level metrics describe how a system behaves over time.

Change reliability asks how a specific modification is likely to affect that system.

For example, a service may have excellent availability while a particular change still has a large potential blast radius.

Both views are useful.

What signals can you measure?

There is no universally accepted formula for software change reliability.

Instead, start with useful evidence.

Change scope

What components, files, APIs, databases, and infrastructure are affected?

Dependencies

How many systems depend on the changed component?

Testing

What evidence exists that the important behavior works?

Production behavior

What do runtime signals tell you about the affected area?

Code volatility

How frequently does this part of the system change?

Incident history

Have similar changes caused problems before?

Deployment outcomes

What happened to similar changes after deployment?

Why one metric is difficult

A single number can hide important context.

Consider two changes:

Change A

Small diff
Shared library
Limited integration testing
Recent incident history

Change B

Large diff
Isolated internal service
Strong integration coverage
Easy rollback

The second diff is larger.

That does not automatically make it less reliable.

The point of measurement is to bring the relevant evidence together.

What makes a useful reliability measure?

A useful measurement should be:

Relevant to the change
Traceable to evidence
Actionable
Comparable over time
Clear about uncertainty

A score is useful when engineers can understand why it looks the way it does.

Where PRI fits

Tomosu's Production Reliability Index (PRI) provides a summary signal across multiple areas of reliability.

The signal can help teams identify which changes deserve closer investigation while keeping the underlying evidence visible.

The main idea

Measuring software change reliability is not about finding one perfect formula.

It is about consistently combining the signals that matter:

Change + dependencies + testing + production behavior + history + deployment outcomes

That gives teams a way to discuss reliability at the level where changes actually happen.

Run a PRI assessment:

https://tomosu.ai/start

DEV Community: Parsa Mohammadi

A Blind Scan Found IBM ContextForge's Connection Leak From One File

The system

What the load test found

Three lines of cleanup

The fix

The experiment: could the code have told us first?

The result

What this does and doesn't show

What AI Reliability Actually Means (and How to Measure It)

The two meanings of AI reliability

Is uptime the same as reliability for an LLM?

How to measure AI reliability

Why agents turn reliability into a change problem

How to set reliability targets for AI features

Common mistakes

Connection Leak vs. Pool Exhaustion: How to Tell the Difference

Leak vs. exhaustion: the definitions

What they look like over a day

Exhaustion in Java, Node.js, Python, and Go

node-postgres waits forever by default

Go's database/sql is unlimited by default

Confirm a leak from the database side

Why leaks show up during a different incident

Code patterns that leak connections

Node.js: client.release() skipped when a query throws

Go: rows or tx left open on an early return

Python: a Session not closed on an exception

Other patterns that pin a connection

Track the borrower when your pool won't

Step by step: leak or exhaustion?

Fixes, matched to the cause

Key takeaways

Change Risk Management in Software Engineering

What is change risk management?

Start before deployment

Use evidence rather than intuition

Match controls to the change

Production reliability and change management

AI assisted development changes the volume

A practical framework

Where PRI fits

The main idea

How to Prioritize Risky Pull Requests

Start with production impact

Look at change scope

Check dependencies

Check production usage

Check testing evidence

Check incident history

Look at runtime behavior

Consider recent activity

Assess blast radius

Consider deployment conditions

A practical prioritization model

Risky does not automatically mean blocked

Where PRI fits

How to Identify High Risk Pull Requests

Start with change scope

Look at dependencies

Check production usage

Check testing

Look at recent activity

Check runtime signals

Check incident history

Assess blast radius

Consider rollback

Large PR does not automatically mean high risk

Where PRI fits

How to Assess the Blast Radius of a Code Change

Start with the changed component

Map direct dependencies

Look at indirect impact

Check production usage

Look at history

Consider the deployment

Diff size is not blast radius

Where PRI fits

Hacktoberfest tip before you submit: break one function on purpose (return []) and run your tests. If the suite still passes, those tests check nothing. Fix them or delete them, a fake green is worse than no test. #hacktoberfest

Agent wrote the code and the tests? Break one function on purpose. If tests pass, they're fake.

Node.js: `client.release()` skipped when a query throws

Go: `rows` or `tx` left open on an early return