A few months ago I thought I was building a linter.
I wasn't.
I was trying to answer one specific question:
Why does so much AI-generated code pass CI and then fail in production?
Not syntax errors. Runtime behavior under load —
event loop blocking, retry storms, lifecycle bugs
that only surface under concurrency.
What I found
I scanned 250 real GitHub repositories.
95,678 Python files. 7,106 findings.
The patterns weren't random. The same anti-patterns appeared
across unrelated codebases, written by different teams,
using different AI tools.
Things like:
-
time.sleep()insideasync def— blocks the event loop - sqlite3 calls inside async handlers — same problem
- retry loops without backoff — retry storms under load
- fire-and-forget coroutines with broken lifecycle
All of it compiled. All of it passed CI.
All of it failed under load.
The part that cost me the most
Not finding the patterns.
Being honest when my own rules were wrong.
One rule started at 95% false positive rate.
I didn't hide it. I measured it, redesigned it,
and marked it publicly as "Limited Scope"
instead of quietly fixing it and moving on.
I think we need more tools that document
when they're wrong — not just when they're right.
What I'm looking for
If you use Claude Code, Cursor, Copilot or Codex daily
and have a story about AI-generated code that escaped to production —
I'd love to hear it.
15-20 minutes. No sales pitch. Pure research.
If you can't share your case, sharing this post helps too.
Landing + full methodology:
https://joaquinriosheredia.github.io/vibeguard-landing/
Repos:
- github.com/Joaquinriosheredia/python-vibe-guard
- github.com/Joaquinriosheredia/java-vibe-guard
Has anyone else noticed this pattern with AI-generated code in production?
python #java #aiengineering #opensource #testing #devtools
Top comments (0)