DEV Community: Junaid Shahid

My AI Agent Found a 5-Sigma Result on Day One. I Deleted It.

Junaid Shahid — Mon, 10 Aug 2026 12:28:32 +0000

I build with coding agents every day. The failure mode people warn you about — the agent hallucinating an API that doesn't exist — is the easy one. Your code won't run, you notice in seconds, you move on.

The failure mode that costs you weeks is the opposite: the agent produces something that looks rigorous and isn't.

Here are two of those, from my own work, and what I changed afterwards.

The first one was easy to catch
I'd built an autonomous research agent: it proposes hypotheses, tests them against historical data, scores them with pre-registered statistics, and buries the ones that fail.

On its very first run it surfaced a result about six times stronger than anything months of manual work had found. Five sigma.

My reaction wasn't excitement, it was suspicion — and the guardrails agreed. The agent had been given access to a data field it shouldn't have had at that point in the loop. It was reading the future. A lookahead bug, not a discovery. Fixed the leak, the "edge" collapsed into noise, and I deleted it the same day.

That one was easy because the result was too good. Five sigma in that domain doesn't happen. The number itself was the tell.

The second one nearly got me
Months later, a much more modest result. Small, plausible, the kind of number you'd actually believe.

I put it through everything I had:

permutation controls — shuffle the labels, confirm the result disappears. It did.
a holdout split — held-out data never touched during development. It held.
tick-level validation — replayed at the finest granularity available, rather than trusting the coarser bars. It survived that too.
Three independent checks, all passed. I was ready to act on it.

It was still an artifact. The execution loop had a bug: when an order failed to fill, the loop rewound and retried it under conditions that had already been revealed. Every fill was slightly luckier than reality allowed. Small enough per trade to look plausible, consistent enough across thousands of trades to look like an edge.

None of my three checks could catch it, because all three ran on top of the same loop. The permutation test shuffled labels and re-ran the same broken loop. The holdout split fed new data into the same broken loop. The tick validation replayed at finer resolution — through the same broken loop.

I'd built the validation with the agent, in the same session, against the same mental model. The validation inherited the blind spot. I retracted the result publicly and wrote up the mechanism.

What I actually changed
Not "trust the agent less." That's not operational advice — it doesn't tell you what to do on Monday.

Pre-register the pass criteria before anything runs. Write down the exact thresholds that would make the result real, and what would make you bury it, before you see a number. Once you have an exciting result, every judgement call bends toward keeping it. This is the single highest-value habit and it costs ten minutes.
Verify at a different layer than the one that produced the result. My three checks all sat on top of the execution loop, so none could see a bug inside it. Now, if an agent builds the pipeline, the verification has to touch reality somewhere the pipeline doesn't — different data path, different tool, different assumptions.
Treat a surprisingly good result as a bug report. It usually is. The prior on "I found something everyone missed" is far lower than the prior on "my measurement is broken." Spend the first hour trying to kill it, not extend it.
Publish the retraction. This is the one people skip. I've since published a result that weakened my own headline number — a metadata heuristic I'd been using over-predicted failures 4:1 against real verification. Publishing that made the original work more credible, not less, because it showed the number had survived someone genuinely attacking it.

The general version
Coding agents are extremely good at producing work that has the shape of rigour. Tests that pass because they assert the current behaviour. Validation that confirms what the pipeline already assumed. Benchmarks measuring the thing that's easy to measure.

None of that is the agent being dishonest. It's optimising toward the target you handed it, and a plausible-looking result is what you asked for.

So the useful question isn't "did the agent get it right?" It's:

What would this look like if it were wrong, and would my checks be able to tell?

If every check runs through the same code path, the answer is no — and you won't find out until it costs you something.

The tools and write-ups from this work are open source at github.com/junaidshahid-dev, including the retraction and the false positives I found in my own auditing tool.

I Installed 300 MCP Servers From PyPI. About 43% Don't Start.

Junaid Shahid — Mon, 10 Aug 2026 12:26:55 +0000

I kept hitting import errors installing MCP servers. Not obscure ones — reference servers from companies whose names you know. After the third, I stopped debugging and started measuring.

The headline: roughly 43% of Python MCP servers on PyPI don't run on a fresh machine. The cause is one character.

Here is the method, because the number means nothing without it.

The trigger
The MCP Python SDK shipped 2.0 and removed several APIs. Any package that declares the SDK with a lower bound and no ceiling —

mcp>=1.0.0
— now resolves to 2.x on a clean install and dies on import. The maintainer's own machine has a working 1.x cached, so pip install never upgrades anything and the bug is invisible to the one person able to fix it. It only breaks for new users.

Stage 1: how many packages are exposed?
PyPI carries about 18,000 packages with "mcp" in the name. I took a seeded random sample of 300 (seed 20260806, so anyone can reproduce the exact draw). Of those, 212 were genuine Python MCP servers rather than clients, docs or forks.

183 of the 212 — 86.3% — declare the MCP SDK with no upper bound.

That is exposure, not breakage. A package can declare an unbounded dependency and still work fine, because it migrated its code and never tightened the pin. Metadata alone cannot tell you. So stage 1 is necessary and not sufficient, and reporting 86% as a failure rate would have been wrong.

Stage 2: install them and see
I drew 30 of those 183 at random and installed each in a genuinely fresh virtualenv — never reused, because a cached SDK is the entire reason this bug hides. Then I found each package's console entry points and ran them with stdin closed.

immediate traceback → broken
exits quietly on EOF, or blocks waiting for input → starts fine
Two failed to install at all. Of the 28 that installed, 14 failed to start.

Combined with stage 1, that puts roughly 43% of Python MCP servers on PyPI in a broken state on a fresh machine, 95% CI 26–61%. The interval is wide because 30 installs is a small sample, and I would rather publish the interval than a tidy number.

Three errors, one cause
ModuleNotFoundError: No module named 'mcp.server.fastmcp'
ImportError: cannot import name 'McpError'
AttributeError: 'Server' object has no attribute 'list_tools'
All three are the same event: code written against 1.x, running on an SDK that removed those APIs.

The part I didn't expect
I assumed the failures would be abandoned weekend projects. They weren't.

Separately from the random sample, I tested 23 established servers — widely used ones, including reference implementations from major companies. 18 of 21 that installed failed to start: 86%, against 50% for randomly drawn packages.

Better-known packages were worse. The likely reason is age. They were written early, against 1.x, and nobody has installed them from scratch since. Popularity doesn't protect you here; it correlates with having been written before the breaking change.

The trap I fell into myself
Having found that unbounded pins predict breakage, the obvious next move is to skip the installs and just read everyone's metadata. It's free and instant.

It doesn't work. I took four organisation-owned repos whose pins flagged them as broken and verified them properly in clean virtualenvs. Only one actually failed. Two started fine despite unbounded pins, and one failed to install for an unrelated reason.

The metadata heuristic over-predicts about 4:1. That is why the tool does real installs, and why I'm publishing this against my own headline number rather than quietly keeping the tidier version.

The fix

mcp>=1.0.0
mcp>=1.0.0,<2 I verified this on every server that failed: pinning below 2.0 restored startup in all of them.

If you maintain a Python package that depends on the MCP SDK, the check takes about four seconds and cannot be done on your dev machine:

python -m venv /tmp/checkme
/tmp/checkme/bin/pip install your-package-name
/tmp/checkme/bin/your-entry-point
A brand new virtualenv, not the one you already have.

Limitations
The install stage is n=30. Hence the 26–61% interval. The metadata stage is n=212 and much tighter.
Tested on Windows with Python 3.13. Resolution can differ across platforms.
"Starts without a traceback" is a low bar. It does not mean the server is correct — only that it runs.
Packages under a separate coordinated-disclosure process were excluded from published results.
Tools and data
Everything is open: the sampler, the verifier, the raw JSON results, and the seed.

https://github.com/junaidshahid-dev/mcp-probe

That repo also lists five false positives I found in my own tool and fixed, because a scanner that fails well-built packages is worse than no scanner at all.

If you maintain an MCP server, go and run the clean-venv check. It takes four seconds and it is not something you can see from where you're standing.