DEV Community

Cover image for AI code is spaghetti. Right?
Jonas Robertsson
Jonas Robertsson

Posted on

AI code is spaghetti. Right?

I built a little tool called Spaghetti Meter to look for spaghetti code in a project. You select a folder, it scans the supported code for problems, and shows what it finds as spaghetti falling into a bowl. The higher the score, the fuller the bowl gets. Around 100 fills it, and after that it starts overflowing.

I thought it would be a fun thing to show other developers. Feed it some AI-generated projects, watch the bowl overflow, have a laugh about the mess we're creating.

I was pretty sure I knew how that would go.

Spaghetti Meter showing spaghetti falling into a bowl during a project scan.

The projects I guessed were mostly AI-written got high scores. But some repos I believed were originally built without AI scored surprisingly high too.

That was the bit I hadn't expected.

That doesn't mean I know who wrote all that code. “Mostly AI” was my impression, not something I'd measured. The same goes for my assumptions about the human-written projects. A contribution policy isn't an authorship record, and these weren't matched projects.

So I went back to the calculation. The first version used findings per file mixed with the total number of findings. I started wondering how much of the score came from the code being messy, and how much came from project size or the way someone had split up their files.

I changed it to use the amount of code actually analyzed instead of file counts. I also reduced how much repeated findings contribute. If the same rule keeps firing in the same file, those findings still count, but they don't keep pushing the score up at the same rate. Naming, style, and security findings still count too.

Those changes made sense to me, but I did make them after seeing results I didn't expect. I'm not going to pretend that gives me a fair comparison all of a sudden.

To see how the updated calculation behaved across different codebases, I ran a small batch of ~25,000 supported files and ~7 million analyzed code lines across twelve projects.

Project AI policy Code lines CB findings Undiminished App score
VoiceStudio Allowed, with conditions 283,220 2,605 159 143
T3 Code Allowed, with conditions 1,081,944 3,044 37 34
Deno Allowed, with conditions 591,608 5,211 237 180
Zed Allowed, with conditions 967,391 6,253 182 132
Bevy Allowed, with conditions 323,930 2,187 192 153
AnyPS5 Allowed, with conditions 730,593 6,289 108 90
QEMU Restricted 1,687,027 6,338 52 46
OpenTTD Restricted 248,109 1,895 99 86
fish Restricted 78,773 1,462 541 410
SDL Restricted 540,598 2,646 65 54
stb Restricted 59,741 417 98 81
Cataclysm-DDA Restricted 663,559 4,775 100 79

The policies are linked, including their conditions. They still aren't proof of who wrote the existing code. CB findings include actionable findings and advisories, not just confirmed bugs.

“Undiminished” counts every repeat like the first finding. “App score” is what Spaghetti Meter actually shows, with repeated findings contributing less. Both use the same analyzed code lines.

What I actually ran

Every project was checked out at the linked commit and scanned twice with the same native CB build:

cb review --advisory --verbose --no-cache --allow-empty --format=json --root /path/to/repo
Enter fullscreen mode Exit fullscreen mode

This was a local development build based on Code Buster 0.14.5. App scores come from the Spaghetti Meter 0.1.5.

Repository settings and normal source classification were respected. Tests, examples, generated and vendored files weren't forced into the scans. No project dependencies or build scripts were run. The repeated scans check consistency, not accuracy or speed; filesystem caches weren't flushed.

I haven't corrected the table by hand or gone through every finding. These are the raw outputs, including the tool's mistakes.

Where the numbers can fool you

The app uses my static analyzer, Code Buster, underneath. If that reports something incorrectly, the spaghetti score inherits the mistake. And even when a finding is technically correct, it might be pointing at something that's perfectly reasonable for that particular project.

Maybe I've built it to prefer standard structures and familiar conventions. And maybe AI is quite good at producing exactly the sort of code my rules like, even if that doesn't make it nicer for a person to read. I don't know. That's something I need to look into, not another conclusion to draw from these scores.

Different languages also get different rule coverage, and the scan doesn't necessarily cover the whole repo. An older project carrying years of compatibility code isn't the same thing as a new project built for one narrow job. There are a lot of things here that a single number won't explain.

I still want to believe human code is easier to read. Maybe it is and my tool isn't measuring the things that make the difference. Or maybe I'm wrong about some of this.

Either way, I thought it was worth sharing rather than only showing results that agreed with me. It wasn't the result I wanted, but it gave me a few things to question about what I'd built.

I'll keep working on Code Buster and Spaghetti Meter, trying to make them faster and sorting out more false positives. For now, please take this as a conversation starter, not something to quote in the next argument about whether AI or humans write better code.

If you want to try it, here's Spaghetti Meter or you could just go with Code Buster instead.

I'd be interested in seeing where it gets your project wrong. Is it finding something genuinely messy, or just complaining about a way of writing code it doesn't understand?

Top comments (0)