I build vibecheck, a CLI that catches what AI coding tools leave behind. Dead scaffolding, invented benchmarks, fake attribution comments, catch blocks that log and swallow. 39 rules, runs in CI.
Last month I added a score. One number for how much of that a codebase carries, per thousand lines, severity weighted, capped so no single rule can dominate. 50 means typical, calibrated against a corpus of real repos.
Then it occurred to me that I had never pointed it at the tools that write the code.
The numbers
Fifteen AI coding tools and agent frameworks, scanned at HEAD on August 6:
| repo | kloc | findings/kloc | score |
|---|---|---|---|
| langchain | 328 | 6.1 | 87 |
| crewAI | 251 | 13.8 | 73 |
| adk-python | 398 | 16.8 | 68 |
| dspy | 58 | 18.6 | 65 |
| litellm | 1864 | 20.2 | 63 |
| AutoGPT | 694 | 20.5 | 62 |
| llama_index | 400 | 23.3 | 58 |
| codex | 47 | 27.1 | 53 |
| OpenHands | 219 | 28.8 | 51 |
| cline | 573 | 37.2 | 42 |
| LibreChat | 763 | 38.6 | 41 |
| gemini-cli | 591 | 39.3 | 40 |
| aider | 34 | 44.0 | 36 |
| autogen | 119 | 44.1 | 36 |
| continue | 276 | 50.3 | 31 |
Median 53. The calibration says 50 is typical. So as a group these are ordinary. Not cleaner than the average codebase, not worse.
I expected that, honestly. The thing I did not expect is the spread.
8x, in the same room
6.1 to 50.3 findings per thousand lines. These projects are doing broadly the same work, in the same window, against the same problem. One of them carries eight times the density of another.
That is the part worth staring at. It means this is not a property of the domain. Nothing about building an agent framework forces you to the bottom of that table. langchain sits at 6.1 across 328,000 lines and 2,536 files, and I checked that number carefully before believing it: 2,538 Python files in the repo, 2,536 scanned, no config suppressing anything. It is just clean.
Where the findings actually land, across all fifteen:
code-quality 52,569 69%
ai-tell 18,842 25%
error-handling 2,412 3%
security 1,847 2%
Two thirds is code quality, and one rule carries a lot of it. no-deep-nesting hit the per-rule cap in 4 of 15 repos, meaning it alone produced more than 20 weighted findings per thousand lines before the cap clipped it. The cap exists precisely so the score does not quietly become a nesting-depth meter.
What this does not measure
It measures what 39 rules detect. That is not the same as quality, and I would not hire or fire a library over it.
Language mix moves it. Python repos trip no-py-print, TypeScript repos trip no-ts-any. A repo is partly being scored on which rules its language exposes it to.
And three of these fifteen (codex, gemini-cli, adk-python) were in the corpus I used to calibrate what "typical" means. So the median being 53 is partly circular. I could have quietly dropped them and shown you a cleaner story. Leaving them in and saying so is more useful than a number I have already tuned to itself.
Run it yourself:
npx @yuvrajangadsingh/vibecheck --score .
The run found a bug in my own tool
I piped each scan into a script to collect the scores. Five of fifteen came back as JSON parse errors.
The offsets were 65536 and 73628. That is not a coincidence, that is a buffer.
stdout to a pipe is asynchronous in node. process.exit() does not wait for it to drain. I had three exits that run after the report is printed, so vibecheck --format json . | jq on a large repo returned truncated, invalid JSON. Silently. Exit code unchanged.
Scanning adk-python:
| output target | bytes | valid |
|---|---|---|
| redirected to a file | 2,685,887 | yes |
| through a pipe | 131,072 | no |
I introduced it three days ago, in a release whose entire purpose was removing silent failures. The exits were added alongside a feature that reports files the scanner could not read, so that people stop getting a clean bill of health for code nobody looked at. That feature shipped with a new way to lose output quietly.
Every one of my CLI tests already runs through spawnSync, which is a pipe. The mechanism was covered. But every fixture produced output small enough to fit in the buffer, so nothing ever had to drain, and the bug sat in a blind spot the tests had walked past dozens of times.
It is fixed in 1.20.1. The new fixture emits 1,800 findings on purpose.
What I take from this
The leaderboard is the less interesting half.
I set out to measure other people's code and the measurement apparatus broke, in a way that had been shipping for three days, in the exact area I had just spent a week hardening. Not because the tests were missing. Because they all sat on one side of a boundary I had never thought about.
If you want to find out where your blind spots are, run your thing against something large and unfamiliar. Fixtures you wrote share your assumptions. A stranger's 400,000-line repo does not.
Top comments (0)