I'm not a programmer. My background is construction engineering, and I got into programming almost by accident: I wanted to understand how Telegram bots worked, so I started stitching one together from pieces of code I found through Google and generated with AI. The result was a monolith I kept breaking — I didn't even know Python cared about indentation, and I genuinely didn't understand why the code kept crashing over whitespace. Through trial and error I slowly figured out where AI tends to get things wrong and how to work around it — with the AI itself helping me figure that out. Eventually I rebuilt the bot from scratch, fully modular this time.
Later I tried Cursor, and it was a good experience — it wrote solid code, even if it sometimes drifted off-task. When my subscription ran out, I went looking for a free alternative and landed on Zed. That's when I wanted Zed to feel as capable as Cursor had — with the same level of understanding of my codebase. So I started figuring out how to build that myself. That's how MSCodeBase Intelligence came to be: an MCP server that gives AI assistants more context about a codebase inside Zed IDE.
I've written about the project itself before. This time I want to talk about something else — a habit I picked up along the way that, in hindsight, matters more than any single feature.
How I actually work
I don't write code by hand. I hand tasks to different AI models — Claude, Gemini, DeepSeek, and others — they write and fix the code, and I verify the result. Not because I'm especially disciplined, but because I genuinely can't just look at code and tell if it's correct. I need something — or someone — to check it.
For a long time that meant: one model writes, I read it myself, and I ask questions when something looks off.
Then I started doing it differently: one model writes a report about what it did or what it found in the code, and a second model — with access to the actual codebase — checks that report against reality. Not because I distrust AI in principle, but because I've learned firsthand that AI can sound completely confident and be completely wrong at the same time.
One specific case
I received an external audit of my project — several pages long, with findings, recommendations, and a list of suspected vulnerabilities. It looked solid. Three findings were flagged as top priority.
I didn't rush to fix anything. Instead, I asked a different model — one with access to the live codebase — to go through the report line by line and check it against the actual repository. Not "does this sound plausible," but literally: open the file, open the line, check whether what the report claims is actually there.
The result was mixed, and that's the interesting part.
Three real findings held up. There genuinely was a process-locking bug on Windows, a case where a data write could be left in a non-atomic state, and a race condition in a task queue that could leave a background task stuck forever. These weren't imaginary risks — they were real, worth fixing.
But alongside that, a few other things surfaced:
- the report referenced a function name that doesn't exist in the code — the model that wrote the audit had likely invented it by analogy, or pulled it from an outdated version;
- one of the file paths in the report was wrong — the actual file lived somewhere else;
- the report mentioned a test that was supposed to catch the bug — no such test existed in the repository at all;
- the test coverage numbers and test counts were stale, from an earlier state of the project;
- for one of the flagged CVEs, the report cited the wrong "fixed in" library version — the actual safe version was later than what was claimed.
So the report was both genuinely useful and partly fabricated — and not fabricated randomly. The invented details were woven around real, correctly identified problems. That, I think, is the most dangerous kind of AI mistake: not "everything is false," but "true, with fabricated details mixed in that you can't tell apart without checking every line yourself."
Why this matters, not just an anecdote
If I'd simply copied the recommendations and asked an agent to apply them, part of the work would have gone nowhere — the agent would have searched for a function that doesn't exist, referenced a test that isn't there, pinned a library to the wrong version. And worse: the three real problems could easily have been dismissed along with everything else, once the fabricated details made the whole report look unreliable.
The lesson I took away: trust in an AI report shouldn't be binary — "believe it" or "don't." A report needs to be broken down into individual claims, and each claim checked separately against the current state of the code, not against the model's memory of what the code used to look like.
A second experiment: checking the instructions, not the code
I keep a file of rules for the AI agents working on this project — something like a job description: how to behave, what to always verify, how to label the source of a claim ("I checked this in the code" vs. "I'm assuming this"). Over time that file grew — a lot.
That raised a question: if I compressed those rules to roughly half the length, would the agent still follow them just as carefully, or would it start cutting corners?
I didn't guess. I ran the same set of tasks twice — once against the full instructions, once against the compressed version — and for each task noted which specific rules were actually followed and which weren't.
The result: the compressed version performed worse, but not dramatically, and not evenly. What suffered most was what I'd call "in-the-moment discipline" — things like updating a decision log right after each step instead of all at once at the end. The rules themselves were still technically present in the compressed file. The agent just remembered to apply them less often mid-task.
That wasn't obvious to me beforehand either. I assumed that if a rule is written down, it gets followed. It turns out what matters is not just whether it's written, but how often — and where in the text — the agent gets reminded of it.
What I'm taking from this
I'm not a programmer, and I don't feel like I understand code better than the models I work with. But in practice I've learned one thing: the main value I add to this process as a human isn't writing code — it's refusing to trust a report about code until it's been checked against reality.
That's slower than just believing it. But the alternative is quietly shipping three real bugs wrapped in two fabricated facts, and never noticing the difference.
I keep a running experiments log directly in the repository — hypothesis, what was actually tested, the raw result, the conclusion, and separately, what I got wrong the first time. Not because it looks nice, but because without it, a month later I wouldn't remember which conclusions were actually verified and which were just stated confidently by something.
The repo is open source, MIT licensed: https://github.com/ManSio/mscodebase-intelligence
If you've run into something similar — an AI audit or report that turned out to be half true, half invented — I'd be curious to hear how you caught it.
Top comments (0)