DEV Community: alphanumericentity

A SELECT that returned more rows than its LIMIT

alphanumericentity — Wed, 30 Sep 2026 10:11:54 +0000

A scheduled job in one of our services started crashing every few days with this:

TypeError: Cannot read properties of undefined (reading 'id')

Inside a for..of over the rows of a query. Not on the first row, not on every run, and never reproducible locally.

The other clue turned up in a log line from the same job, which prints how many pending records it picked up. The query has LIMIT 500. The log said pendingTotal: 534. Another run said 813.

A SELECT with a LIMIT of 500 returning 813 rows is the kind of thing that makes you go back and read your own SQL four times. The SQL was fine. So was the ORM. The bug was one line below the driver's surface, and it has been open upstream for about a year.

What actually happens

We use postgres-js (the postgres package) with a connection pool, like most people do. Here's the shape of the bug, straight out of src/connection.js in the current release:

A connection keeps a single write cursor for the result it's filling in:

let ...
  , rows = 0

Every DataRow message the server sends gets written at that cursor:

function DataRow(x) {
  // ...
  : (result[rows++] = transform.row.from ? transform.row.from(row) : row)
}

rows goes back to zero in exactly two places: CommandComplete, which is the server saying the query finished, and PortalSuspended, which is the cursor case. That's it.

Now look at what happens when a query doesn't finish. ErrorResponse records the error and returns. Then ReadyForQuery cleans up:

function ReadyForQuery(x) {
  // ... resolve or reject the query ...
  query = results = errorResponse = null
  result = new Result()
  connectTimer.cancel()
}

It allocates a fresh result array. One line later than where the cursor should have been reset, and the cursor isn't in that list.

So: a query streams 97 rows, then the server kills it. rows is left at 97. The connection goes back in the pool. The next query that lands on it gets a brand new empty array, writes its two rows at index 97 and 98, and resolves with an array whose length is 99 and whose first 97 slots are holes.

Why this one is nastier than a normal wrong answer

A sparse array is a bad failure mode because nothing about it throws:

JSON.stringify(rows) gives you a list starting with 97 nulls. That goes straight into a log line or an API response.
for..of and .entries() hand you undefined, which is where our TypeError came from.
.length is 99, which is how a LIMIT 500 query reports 534.
.map() preserves the holes, so an ORM mapping layer passes them through.
.filter(Boolean) hides it completely, so whether you ever find out depends on which array method you happened to use.

And the quiet one, which is the reason I'm writing this down:

const [row] = await db.select().from(users).where(eq(users.id, id));
if (!row) return notFound();

row is undefined, the record exists, and you take the "not found" branch. No error, no log, just a wrong decision. We had a version of that on a write path.

The trigger is almost always a statement timeout

To hit this you need a query that errors after it has streamed at least one row. The common way is statement_timeout, Postgres error 57014, firing mid-scan on a query that's already returning rows.

The thing to check is where your timeout is set. Ours is on the database role, not in application code, which means any slow query in any service arms this for the next query on that connection. One heavy scan in a cron job, and an unrelated request handler sharing the pool gets the holes. That's also why it never reproduced locally: on a laptop nothing takes long enough to get cancelled.

If you want to confirm it in your own logs, look for these two together:

A count in a log line that exceeds the LIMIT of the query it came from.
A TypeError on undefined inside a loop over query results, within a few seconds of a canceling statement due to statement timeout in the same process.

Same process matters, since the pool lives in the process. Different pod, different pool, unrelated.

Upgrading doesn't fix it

I checked the current release and master while writing this. postgres@3.4.9 is the latest on npm, and both it and master reset the cursor in the same two places and no others. The fix is a one-line PR, #1120, open since October 2025. The mechanism is written up in issue #1181, which is open and still getting comments this week.

So until it lands, patch it. With pnpm:

pnpm patch postgres

Add the one line to ReadyForQuery:

   query = results = errorResponse = null
   result = new Result()
+  rows = 0
   connectTimer.cancel()

Then pnpm patch-commit <dir>, which writes the patch file and the patchedDependencies entry into your workspace. npm and yarn both have their own equivalents.

If you'd rather not patch a dependency, the other mitigations are all worse but worth knowing: set max: 1 so a poisoned connection only hurts the code that poisoned it (it doesn't make the problem go away), drop your statement_timeout low enough that queries die before streaming anything (good luck), or check rows.length against your own LIMIT on every read and throw. We patched.

The general lesson I took

I had a rule in my head that an error on a connection is contained: the transaction rolls back, the pool hands the connection to someone else, life continues. This was a reminder that the client library also has per-connection state, and an error path is exactly where that state stops being maintained.

So when a client hands you a fresh object for each query, the question worth asking is whether anything else is still pointing into the old one.

Give every coding agent its own git worktree

alphanumericentity — Wed, 16 Sep 2026 02:58:28 +0000

Quick scenario. You've got two coding agents going in two terminals. Agent one is halfway through a refactor. Agent two gets asked to fix a bug that lives on another branch, so it does the obvious thing and runs git checkout. Agent one's next tool call reads files that aren't there anymore. Neither of them notices. Both keep going.

I've had some version of this happen more than once before I stopped and fixed it properly. The problem is that a git checkout was designed for one person with one intention at a time. You'd never switch branches while a coworker was typing in the same directory, but an agent has no concept of "someone else is using this". It assumes the tree it saw a minute ago is the tree it has now. Often that's just false.

The fix turned out to be a git feature I'd basically ignored for ten years: worktrees. They've been in git since 2.5 (2015). One repo, multiple working directories, each on a seperate branch, all sharing the same object store. One human rarely needs two checkouts of the same repo so nobody bothered. A handful of agents need exactly that.

Ok so what's a worktree

git fetch origin main
git worktree add -b fix/login-redirect ../myrepo-wt-login-redirect origin/main

That makes a branch off the current origin/main, checks it out in a sibling directory, and registers it with the repo. Leave off the -b if the branch already exists. git worktree list shows you all of them, git worktree remove ../myrepo-wt-login-redirect gets rid of one.

Same .git, many directories. A branch can only be checked out in one worktree at a time, which sounds like a limitation but is actually the whole point. The second agent that tries to grab a branch gets an error instead of silently pulling the rug out.

One thing: put the worktree next to the repo, not inside it. If it's inside, the main checkout sees a big new untracked directory and sooner or later someone's git add . swallows it, and that's a fun one to untangle.

That's the mechanism. The mechanism alone doesn't save you though, the rules around it do.

The four rules

These live in the agent's instruction file (CLAUDE.md, AGENTS.md, whatever your tool reads). Not in my head, because I forget, and the agent definitely doesn't read my head.

Isolate. Every edit happens in a dedicated worktree created off the exact base you mean. New change? Fetch and base it off origin/main. Fixing an open PR? Base it off that PR's current remote head, not whatever stale local copy of the branch happens to be lying around. The main checkout is read only for agents. Grep in it, git log in it, don't edit in it.
Stage by path. git add -A and git add . are banned. The tree might have another agent's half done work in it, or a stray .env, or some debug script. An agent that stages everything commits everything. It names the files it touched and those go in, nothing else.
Check the diff. Twice. Before pushing, git diff origin/main...HEAD (three dots, not two, this matters) has to be exactly the change you intended, nothing extra pulled in and nothing missing. Then again before merging, because main moved while the agent was working and the diff that got reviewed at push time is not the diff that actually lands. If anything unexpected shows up the agent stops and says so. It does not merge and hope.
Clean up. git worktree remove once the PR is merged, then delete the branch. Old worktrees that outlive their PR are exactly where the next agent finds a stale branch and builds on top of it.

Here's roughly what's in my instructions file:

## Changing code
- Never edit in the main checkout. Work in a worktree off the intended base:
  git -C /abs/path/to/repo fetch origin main
  git -C /abs/path/to/repo worktree add -b <branch> /abs/path/to/repo-wt-<slug> origin/main
- Stage by path. `git add -A` and `git add .` are forbidden.
- Before pushing, and again before merging, `git diff origin/main...HEAD`
  must be exactly the intended change. Anything else: stop and report.
- Update a pushed branch by merging main into it, never by rebasing.
- Remove the worktree once the PR is merged.

Two things in there are on purpose.

Absolute paths and git -C everywhere. An agent's working directory is not something you want to depend on. Some harnesses reset it between tool calls, some (Claude Code, in my setup) prompt you every time a command starts with cd, and an agent that thinks it's in the worktree but is actually sitting in the main checkout will do precisely the thing the worktree was supposed to prevent. Absolute paths make that whole class of bug impossible.

And merge, don't rebase, once a branch has been pushed. A rebase rewrites commits that already exist on the remote, then needs a force push, and the force push throws away anything a human added to the branch in the meantime. An "update branch" click, a review fix, a commit from a coworker who had no idea an agent was on that branch. I know rebasing gives you a prettier history. I don't care anymore.

Why not just use branches?

Fair question, git already has branches.

Because switching a branch is a write to a shared mutable global. Every agent in that checkout sees it and none of them get told. It also trashes everything in the tree that's keyed to its contents: build output, the bundler cache, tsc's incremental state, node_modules if the branches have different deps. Two agents ping ponging between two branches in one directory spend most of their time rebuilding the world and very little of it doing the task.

Also you can't run two dev servers from two branches out of one directory. With worktrees you can, and you'll want to, because "does the fix actually work" means running the thing.

Stuff that will bite you

The mechanism is simple. The edges are where the afternoon goes.

Untracked files don't come along. A new worktree is a clean checkout of tracked files only. Your .env, local config, that cert you generated months ago, none of it is there. Copy what the tree needs over explicitly. And don't let the agent "fix" a missing .env by committing one.

node_modules is per worktree. Every tree needs its own install. With pnpm this is nearly free, packages get hardlinked out of the global content addressable store so a second install is mostly bookkeeping. With npm it's a full copy every time, which adds up fast. If your monorepo has a build cache, the local part of it is per tree as well, so a remote cache is what makes worktree number two start fast rather than rebuild everything.

Ports collide. Two worktrees, same app, same port. Derive the port from the worktree name or pass it explicitly, and make the agent print the URL it actually ended up on.

The database is the next shared global. Two agents, two worktrees, one local Postgres. The moment one of them runs a migration the other's app is talking to a schema it doesn't expect. Per branch databases (a database or schema named after the branch, created on first boot) close this hole. It's the same move as the worktree, one level down.

Hooks and config are shared. Every worktree reads the same .git/hooks and the same repo config. Good, the guard rails apply everywhere. But a hook that assumes it's running from the main checkout path will do weird things in a worktree. Use git rev-parse --show-toplevel in hooks, never a hardcoded path.

Stashes are shared too. The stash is a ref on the repo, not the worktree. An agent that stashes in one tree leaves a lovely surprise for whoever runs git stash pop in another. Honestly just don't stash. Commit the WIP on the branch and squash it later, or don't, nobody cares.

Lockfiles diverge. Two worktrees each add a dependency and now the lockfile conflicts at merge time. Never hand merge a lockfile. Take either side, run install, commit whatever it produces.

Strays pile up. If you rm -rf a worktree directory git still thinks it's there. git worktree prune clears the bookkeeping. git worktree list is the first thing I run when anything git related is being weird. A branch that's checked out in some worktree also can't be deleted or checked out anywhere else until that worktree is gone, which is the first thing to check when an agent complains that a branch is "already checked out".

What actually changed

Before, the checkout was a global. Everything in my agent setup had to reason about its state and none of it could, because none of them could see the others.

Now the checkout is a parameter. Each agent gets its own, does its work there, proves what it did with a diff against the base, and the directory goes away when the PR merges. The main checkout just sits on main, clean, and I mostly use it to read code.

None of this is clever. It's one git command that's been sitting there for a decade plus four rules written down somewhere the agent will actually read them. Most of the damage agents were doing to each other's work just stopped, and whatever's left at least shows up in the diff.