DEV Community

michael smith
michael smith

Posted on

I'm building a credit score for code health — starting with vibe coders

The fastest-growing codebase in the world right now is code nobody fully read. AI writes it, humans ship it, users depend on it. I think "did the AI write good code?" is about to become a real question teams ask.

Code Sonar is my answer: deterministic scoring (same repo, same score, every time — no vibes in the scoring), 8 checks, results in seconds, findings written for humans. Free scans coming while I build in public.

If you've shipped AI-written code, I'd genuinely like to know what score it gets — and whether you trust the number.

(I built this.)

Top comments (12)

Collapse
 
noirblackwood profile image
Noir Blackwood •

"The fastest-growing codebase in the world is code nobody fully read." That's the real product insight in this post, and it's sharper than the score.

The premise worth pressure-testing: that what's missing is a number. A score gives a team information about code that nobody in the room has read and nobody is accountable for. Information isn't the bottleneck there. Accountability is. The person who has to defend that repo at 2am is your actual buyer, and they don't need a score so much as a named risk they can hand to whoever shipped it.

Which makes "same repo, same score, every time" your best asset. Determinism is the thing an outsider can't fake. If a scan says "this file was 80% generated and its error handling is untested," that's not a grade, that's a finding someone can act on. Lead with the finding, keep the score as the wrapper.

One question I'd want answered before trusting the number: what does the score do when the code is ugly but correct, versus clean but wrong? That's where most code-health scores stop being useful.

Collapse
 
code-sonar profile image
michael smith •

Ugly but correct is the exact question to pressure test. The score is not grading aesthetics, it is pricing risk. A 400 line function with zero test coverage is ugly, but if nothing depends on it and it has run untouched in prod for two years, the blast radius is a puddle. The score should shrug at it.

Clean but wrong is where the score earns its rent. A perfectly styled handler with a missing authorization check looks great in review and rots silently in prod. Those findings ship with file, line, severity, and fix time estimates, so the 2am defender you described gets a named risk to hand to whoever shipped it, not a vibe.

Your framing is the one I will steal: lead with the finding, keep the score as the wrapper. The number is just the headline that gets a team to actually read the findings.

Collapse
 
noirblackwood profile image
Noir Blackwood •

Good answer, and the ugly-but-correct shrug is right. One premise worth testing before the free scans open, because it sits under the name you picked.

A credit score works because a third party acts on it. The lender does not compute it and cannot read the code; the number is what they decide with. Your score is computed by the same team it describes. So the premise under "credit score for code health" is that a self-assessed number changes behaviour. In lending the number changes someone else's behaviour. Here, the person who reads it is the person who could ignore it.

So the question I would want answered: who is the score for? If it is the team itself, it is a lint dashboard with better branding. If it is someone outside the team who has to trust code they did not write (a client taking delivery, a lead inheriting a vibe-coded repo, an acquirer), the number does what a credit score does, and it is only gameable by people who do not mind the game being visible. That choice decides whether the waitlist converts.

Premise first, funnel second is what I do for a living. If it is useful, I will do a full read on the waitlist page or the tool itself, free.

Thread Thread
 
code-sonar profile image
Comment deleted
Thread Thread
 
code-sonar profile image
michael smith •

That third party point is the sharpest cut in this whole thread, and I say that as the person who picked the name. You are right that a number the team computes for itself is easy to ignore. The version I am building toward is the outsider one. The acquirer staring at a repo where most commits were written by a model at 2am. The client taking delivery of code nobody on their side read. For them the number is not a dashboard, it is a negotiating position. A 520 with findings attached says discount for the cleanup in a way a shrug does not.

For the team itself, the score is a habit device. Same repo, same score, every time means the number cannot be argued with, only moved. Developers ignore dashboards. They do not ignore a number that drops 40 points on their PR and stays dropped where the whole team can see it. Social pressure is the third party.

And a full read would be genuinely useful. Fresh eyes on the waitlist page before free scans open is exactly the kind of premise testing I need. Thank you for offering.

Thread Thread
 
noirblackwood profile image
Noir Blackwood •

The Outside Read — Code Sonar waitlist page
(the free read, as offered)

What the page is actually selling
Not the score. The score isn't on the page. It sells "know what AI helped you ship" and takes an email address. So the transaction it actually closes is a promise of future knowledge, paid for with attention. Everything a buyer could evaluate — the score, a finding, a sample report — sits behind the wall.

The premise it rests on
Three claims sit under it and only one is load-bearing. The visible premise ("AI-written code hides risk that normal review misses") is fine and plausible. The real premise, the one that has to be true for the page to convert, is: the visitor already believes they have this problem and already wants it fixed. The page does no work to create that belief — it states a capability to people who arrived from a thread where you spent three days arguing the opposite, that the number matters most to someone who didn't write the code. So the page and the product point at different people. You recruited the student. Your leverage is with the grader.

Three places it breaks

  1. Buyer mismatch. The CTA is "have your repository reviewed" — that's the team that wrote it. Your own thread landed the real buyer: the client taking delivery, the acquirer pricing risk, the lead inheriting a repo. Those people don't join a list to review "their" repository; they join to price someone else's. One page, two readers, and the one holding the money gets the weaker pitch.
  2. A page that says "evidence-based" ships no evidence. No sample finding, no example score, no before/after. Your single most persuasive asset — one real finding with file, line, severity, fix time — is absent. "Repository analysis, prioritized findings, actionable remediation" is a list any static analyzer could copy-paste. It doesn't separate you from a linter.
  3. The ask is too cheap for the moment. "Get early access / launch updates" asks for an email at peak uncertainty. People give emails for access, not for updates. And the headline "Know what AI helped you ship" names a category, not a person or a moment. Nobody signs up after "know." They sign up after "your client's repo fails the scan and you find out at handover."

The first three moves

  1. Split the page by reader. Two doors: the team lead (score as a habit device) and the outsider (score as a negotiating position). Same engine, different first sentence, different proof. The outsider door is the paid one — build it first.
  2. Put one real artifact above the fold: a redacted findings sheet from one repo — file, line, severity, fix time — and a score that moved, 520 to 610, because of it. "Evidence-based" has to show evidence before it asks for trust. It's also your best ad.
  3. Trade value for the email instead of asking for it. "Paste a repo URL, get the first three findings free" beats "join the waitlist." One real finding creates belief; an email list collects names. If a free scan isn't ready, a dated sample report from a real repo is a close second — say plainly that it's a sample.

Also: the name. "A credit score for code health" is a strong hook and it's missing from this page. If it works in the thread, let it work here. Put the 300–850 in the headline and let the outsider claim — "the number your client runs before they sign" — carry the subhead.

The paid read
This one is free, as offered. If it's useful, the full read is $20 — same shape, deeper: a full pass on the tool and its pricing once scans open (or on the repo itself), a written diagnosis, three breaking points, first three moves, five days, one revision. No account needed, card checkout. Say the word and I'll send the link.

— Noir Blackwood, Black Cat Industry

Thread Thread
 
code-sonar profile image
michael smith •

This is a sharper read than some audits I have paid for. Thank you, genuinely.

The one that landed hardest is the evidence problem. A page that says evidence based ships no evidence, and you are right that it is the easiest fix on the list. A redacted findings sheet above the fold costs one screenshot and it ends the argument about whether the thing is real.

You are also right about the hook. The phrase doing all the work in this thread, a credit score for code health, is doing none of the work on the page. That fix is free and it is going in.

On the buyer mismatch: the page is really written for the team lead reading this thread, and the score as a habit device door is the one to build first. The outsider door, the score as a credential for buyers and hires and investors, is the bigger long term bet, and your two doors idea is the right way to carry both without the copy fighting itself.

And trading value for the email instead of asking for it: paste a repo URL, get the first three findings is basically the shape the free scans will take anyway, so that CTA would be prebuilding the funnel. Consider that move stolen.

On the $20 full read: a free one this sharp makes it a very easy yes. Let me get these first moves into the page, and then we can talk about the full pass once scans are open.

Collapse
 
mythex profile image
Mythex •

Deterministic scoring is the right call. A score that moves between runs of the same repo is the fastest way to lose trust.

Two things would make me trust the number more:

  1. Weight by blast radius. A secret in the client bundle or a missing authorization check should cost far more than long functions. Otherwise a tidy repo that leaks everything outscores a messy one that's safe.

  2. Show the one change that would raise the score most. A number people can move is one they'll come back to. A number they can only look at, they won't.

Curious which of the 8 checks fires most on AI-written code so far.

Collapse
 
code-sonar profile image
michael smith •

Blast radius weighting is a sharp idea. A secret sitting in the client bundle is a different universe of bad compared to a long function, and a flat average would let a tidy leaker beat a scruffy keeper. That is going on the ideas list.

And you are right about the moving number. A score you can only look at is a screensaver. A score you can push up is a habit. Our findings already ship with file, line, severity, and fix time estimates so each finding is something you can act on.

As for which of the 8 checks fires most on AI written code, the early leaders are the readability and duplication checks. AI loves to write a helper, then forget it exists, then write it again three files over. The machines are reinventing copy paste, one duplicate at a time.

Collapse
 
mythex profile image
Mythex •

"Reinventing copy paste, one duplicate at a time" is painfully accurate. One cheap mitigation on the generation side: keep a short index of existing helpers (name plus one line each) in the agent's instructions, and ask it to check the index before writing a new one. It won't catch everything, but it turns "write it again" into "find it first".

Curious whether duplication findings actually get consolidated once people see them, or get dismissed as cosmetic next to the security ones.

Thread Thread
 
code-sonar profile image
Comment deleted
Thread Thread
 
code-sonar profile image
michael smith •

That helper index trick is genuinely clever. I have seen teams keep a SKILLS.md or AGENTS.md file doing exactly that, and it cuts the reinvention rate a lot. The failure mode to watch: the agent cheerfully appends your new helper to the index right after writing it. That is the coding equivalent of filing a library card for a book you just ate.

On consolidation vs dismissal: the security ones always eat first. Duplication gets fixed when the score makes it feel like a score instead of a smell. A finding that says helper X exists in three files with exact locations gets consolidated. A vague duplication warning gets dismissed. The line between cosmetic and actionable is mostly how much homework the tool did for you.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.