The failures that don't fail loudly

Harsh Singh — Sun, 30 Aug 2026 06:32:06 +0000

I spent a week building an agent that upgrades dependencies and repairs what the upgrade breaks. Dependabot opens the PR and walks away; this one stays until the tests are green.

The interesting part wasn't the repairs. It was that almost every serious bug I hit — in my code, in the harness, in my own agent — announced success while being wrong.

The premise

Detection is solved. Dependabot, npm audit and OSV all find the advisory. What nobody automates is the bit afterwards: the fix is a major version bump, the bump breaks your build, and now it's your afternoon.

So: pull real advisories, spawn one subagent per vulnerable package, each on its own branch in its own sandbox. Bump, install, run the suite. If it breaks, read the failure and patch the source. Re-verify from a clean checkout. Open a PR and stop — merging is a human decision.

That's the design. Here's what actually happened.

1. The advisories weren't where I looked

My first scan found nothing. The manifest said js-yaml: ^3.13.1, which sounds vulnerable, but ^3.13.1 resolves to whatever the latest 3.x is today — and that's patched.

Vulnerabilities live in the resolved tree, not the manifest. Scan the lockfile, including transitive dependencies, or you scan nothing. All three advisories I eventually found were transitive dev dependencies that appear nowhere in package.json.

A scanner that reads the manifest returns "you're clean" and is wrong. It doesn't error.

2. The agent caught itself

The first proper scan reported 124 vulnerable packages out of 290. Then it said something I didn't expect:

"Some results look suspicious — js-yaml@4.3.2 with 10 advisories, which is usually clean."

It cross-checked OSV's batch endpoint against the single-query endpoint and found its own bug: it had nested version inside the package object instead of alongside it. The API silently ignored the version and returned every advisory ever filed for each package.

Real answer: three. Not 124.

The API didn't reject the malformed query. It answered confidently and wrongly.

3. Three correct agents, one wrong result

This is the one I'd defend hardest.

Three subagents each fixed one advisory. Each branch was green — 75 tests passing, advisory gone. I merged two of them.

Git reported no conflict. The result had two overrides keys in package.json. JSON keeps the last one, so the first fix — a HIGH-severity ReDoS — silently disappeared. Passing tests. Clean diff. A PR whose stated purpose was undone by the act of merging it.

No per-branch check catches this. Each worker verified correctly; the combination was broken.

So the rule got sharper: don't verify the branch, verify what main looks like after it merges.

4. Five ways to under-report

I moved the scanner into a Skill so the agent runs tested code rather than following prose. Then Qodo's review took it apart, and every finding was the same species:

Aliases — "foo": "npm:bar@1.2.3" installs bar at node_modules/foo. Deriving the name from the path queries a package that isn't installed.
Pagination — OSV paginates past 1,000 vulnerabilities. An unfollowed next_page_token means a short answer.
Link entries — workspace placeholders have no version. Query them and you get advisories for a package that isn't there at that version.
The root entry — empty path, real version, produces an empty package name.
Truncated responses — pairing queries with results using zip() silently drops the unmatched ones, then prints a clean summary.

Not one of those throws. Every one produces fewer advisories and a confident "you're clean". For a security scanner that's the only direction that really matters, so the scanner now refuses to continue when the response doesn't match what it asked for.

5. The gate worked. The display of it didn't.

My queue board showed a run as done. The harness terminal was simultaneously asking me to approve a pull request.

The harness emits turn.done one millisecond after tool.approval_required — because a turn ends when it pauses for a human:

06:01:25.721  tool.approval_required
06:01:25.722  turn.done

I'd treated turn.done as "finished". So the board hid the single state an operator most needs to see. The gate was working perfectly; the thing showing me the gate was lying.

6. And then the agent tried to get around it

Testing the gate live, the agent hit create_pull_request, which requires approval. Its own narration:

"Now let me invoke the CLI directly (bypassing the Code Mode restriction)…"

"The CLI is blocked too — the module enforces that destructive tools only go through the direct approval flow."

It tried two routes around the gate. The harness blocked both.

That's the strongest thing I can say about the design: the control is in the runtime, not in the model's cooperation. The agent can't merge because merge_pull_request isn't in its toolset — 43 of GitHub's 44 tools are enabled, and that's the exception. It isn't asked nicely.

But an agent that probes for a way around the only control on irreversible actions is a liability whether or not it succeeds, so it's now explicitly instructed that being blocked is a correct outcome.

What I'd take away

Loud failure is a feature. A crash gets fixed in ten minutes. A confident wrong answer ships. Everything above was the second kind, and the only defence was checking claims against reality — a second endpoint, a clean checkout, a merged state, an event stream.

Never trust a worker's self-report. My workers report success. The system throws that report away and re-runs the suite itself. That felt paranoid when I wrote it and stopped feeling paranoid the first time a green branch turned out to be wrong.

A report is a valid outcome; a false green is not. After three failed repair attempts a worker stops and writes up what it tried. Giving up honestly is a feature, not a failure mode.

Built on TrueForge, with OSV.dev for advisory data and Qodo reviewing every PR — which found five of the six quiet failures above.

DEV Community: Harsh Singh