We refuse to install a dependency without a lockfile, a checksum, and a license audit. Then we paste a language model's answer straight into a model file and call it velocity. The scrutiny didn't disappear — it just stopped applying to the thing writing the code.
The paradox
We have built an extraordinary apparatus of distrust around our own work. Lockfiles pin transitive dependencies to exact versions. Checksums verify the bytes arrived intact. Signed commits prove authorship. CI refuses a merge over one failing test. Code review exists because we decided, collectively, that no engineer's confidence in their own code is sufficient evidence that the code is correct.
Then a model produces forty lines, and they go in.
Not because anyone sat down and decided generated code deserves less scrutiny. It's that every mechanism we built assumes a human author on the other end — someone who can be asked why, who hedges when unsure, who owns the thing afterwards. Generated code walks past all of it, because none of it was pointed at that door.
The standards haven't changed. The risk hasn't changed. Only one thing has: the single participant in the process that cannot be held responsible for being wrong is now also the one we check the least.
As a software engineer, that asymmetry tells me the real disagreement isn't about whether AI writes good code. It's about what we are willing to call checked.
Hallucination isn't the interesting question
Most debates about AI and code ask whether the model hallucinates. Fair enough — it does. But hallucination is the easy failure, because it announces itself. An invented API doesn't exist. The import fails, the type checker complains, the test goes red. The system you already have catches it.
The failure that matters is quieter.
During development, my 7B cited odoo.onchange-is-not-validation — a real rule, in my corpus, traceable to upstream documentation — and then wrote an @api.onchange to guard the value. Every citation verified. The answer was still wrong.
Citation existence is not rule compliance. That one sentence is what the whole project is built on.
I've come to think of the underlying machine as a fluency machine. I use the word deliberately. It doesn't manufacture wrong answers — wrong answers predate it by the entire history of the profession. What it manufactures is the appearance of having checked. A junior engineer who is unsure hedges, and you can hear it in the hedge. A model that is unsure produces the same fluent, well-structured, confidently-cited paragraph as a model that is right. The uncertainty signal we spent careers learning to read simply isn't in the output.
Fluency is not a confidence interval. It just renders like one.
A prompt is not a specification
Here's where software engineering has something to say. Our entire discipline runs on abstraction — interfaces, APIs, frameworks, platforms — each layer letting one engineer reason about one problem without drowning in all the others. A prompt is an abstraction too. It compresses everything you know about a domain into prose and hands it over as though prose were a contract.
But every abstraction hides what it discarded, and prose hides more than most, because prose has no schema. You can't diff it meaningfully, you can't validate it, and you cannot test whether a paragraph still means what it meant before you edited it.
I found the edge of that the ordinary way. Prompt tuning plateaued after two iterations. Hardening the Odoo rule about returning false for an empty field fixed the scalar cases and simultaneously broke the Many2one tuple handling that had been correct the day before. A 7B holds a bounded number of constraints; past that point, more instruction text just trades one error for another.
The same limit showed up in routing. My first attempt used a single system prompt covering both Odoo and TypeScript. Asked a plainly Odoo question — cap a sale order line quantity — it answered entirely in TypeScript with a Zod schema. The instructions bleed. A small model doesn't reliably route between two stacks described in one paragraph.
So I moved the rules out of the prompt and into data. Each rule lives in corpus/rules/*.json against a JSON Schema: a citable id, a severity, the failure it prevents, a wrong/correct code pair, an upstream documentation URL where one exists, and where possible a regex that detects the violation in generated code. The system prompts and the retrieval index became build artifacts, generated from that corpus by nexa build. Same principle as z.infer<typeof Schema> — one definition, derived artifacts, no drift.
That didn't make the model smarter. It changed the failure mode. Instead of rewriting prose and hoping, a rule is written once and flows to the prompt, the retrieval index, and the compliance checker together.
Nobody misconfigured it, but the system decayed
Distributed systems engineers know the shape of the next part. A latency bug in one service cascades through a queue, triggers retries, saturates a database, and surfaces as an outage somewhere else entirely. Every component behaved to spec. The system as a whole did something nobody designed.
I benchmarked the 7B at 37 tok/s. Four instrumented runs later, the report showed roughly 10. I offered three diagnoses; the first two were wrong.
The actual cause was that Ollama's memory estimator is conservative and leaves layers on the CPU with VRAM still free — 11.7 tok/s automatic versus 38.6 forced, same model, same context. My Modelfiles carried num_gpu 99, but the pipeline was calling the raw base model, which has no such parameter. Meanwhile the KV cache environment variables had been set while the server was already running, so they never took effect at all. And two orphaned llama-server processes were sitting there holding 1.4 GB of VRAM, left behind when their parent was killed.
Nobody misconfigured anything. Ollama's estimator was being careful. Windows was doing what it does with environment variables set after a process starts. The orphans were doing what orphans do. Collectively, the system ran at roughly a third of its speed for most of a working session — while I read a number I had measured once and believed ever since.
An unmeasured number decays into a claim.
That is the entire reason nexa doctor exists, and why it checks GPU residency and each of those traps by name rather than just confirming that Ollama is up. There's an old rule in operations: someone owns every service. Complexity raises the effort of attribution. It never removes the need for it.
Patterns inherit their author's blind spots
You can't check for "is this code correct" — the phrase is too vague to be an objective. So, exactly as we do everywhere in engineering, we check proxies. A regex matching @api.onchange in a validation context is a proxy for "this code is using the wrong mechanism to hold an invariant."
Individually, these are reasonable. The problem is that a proxy always discards information, and it quietly inherits the assumptions of whoever wrote it.
So the patterns get tested too. Every rule already carries a wrong and a correct sample, which means the test data existed for free. nexa validate fails the build on three faults: a pattern that doesn't match its own wrong sample catches nothing; a pattern that fires on its own correct sample cries wolf; a satisfied_pattern that clears its own wrong sample is inert by construction.
Then nexa crosscheck fires every pattern at every other rule's correct sample — because self-testing proves a pattern handles its own case and says nothing about whether it goes off on unrelated good code. The first run found three real defects. float-comparison fired on qty > 0, including inside SQL where float_compare doesn't apply; narrowed to equality, which is the actual trap. odoo-read-returns-array fired on any Schema.parse(await ...). And ondelete-is-a-decision correctly flagged a sample in my own corpus that didn't follow its own rule.
That last one is the one I'd keep. The checker was right and the corpus was wrong.
It's also the loop worth being nervous about: my rules become the system prompt, the system prompt shapes the answer, and the answer is then checked against the same rules that produced it. A rule that is subtly wrong is wrong in three places simultaneously, and the pipeline will report VERIFIED at every step. All models are wrong but some are useful — the danger was never the simplification. It's forgetting what got discarded in it.
Which is why the limitation goes in the docs, in plain words: 27 of 32 rules trace to upstream documentation, and the rest rest on general knowledge and deserve more scepticism. These rules were written by an assistant, not harvested from production incidents. Rules drawn from real incidents would be worth more.
"Verified" can be theatre
The standard reassurance in this space is that the tool validates its own output. From an engineering perspective, that tells you almost nothing about how the system behaves. Inserting a step into a pipeline doesn't guarantee verification, any more than a deployment approval guarantees oversight — an approval only matters if the reviewer has the information to spot risk, the authority to say no, and the time to actually look.
We have a name for the human half of this: automation bias. Once software earns a reputation for being usually right, reviewers start confirming its output instead of interrogating it.
The checker half has a sharper version of the same problem. A checker that fires on good code is worse than no checker, because it teaches you to dismiss it — and once you're dismissing it, it will be dismissed on the day it was right. So five of my rules deliberately carry no pattern at all, with the reason recorded for each. Two need set comparison between a decorator's arguments and the attributes its body reads, which is AST work, not regex work. One lives in a diff between module versions. Two depend on context a snippet doesn't contain. Those rules return no verdict rather than a false pass. 84% coverage with an honest list of the gap beats 100% with five patterns producing noise.
So "did it verify?" is the wrong question. The right one is: what exactly was checked, and what does the absence of a finding actually prove?
In Nexa, VERIFIED means the checks that exist found nothing. It does not mean correct. Compliance is only checked against rules retrieved for that question, plus pinned ones — so a rule that was never retrieved is a rule that was never enforced. I'd rather ship that sentence than a green checkmark.
Small and local is not a lesser tool
Now flip the paradox around.
A 7B-to-30B local model does not match a frontier model on hard architectural reasoning. That gap is a function of scale, and no amount of prompting closes it. I put that in the README, near the top, deliberately.
But engineers make a distinction when optimizing: removing overhead is not the same as increasing capability. Stripping out overhead lets existing resources do more — it doesn't upgrade the hardware. Nexa doesn't make a small model smarter. It removes the thing that made a small model untrustworthy: the gap between an answer and any means of checking it.
And for this particular job, that trade isn't obviously a loss. The frontier model has more capability. The local one has more provenance — it answers from rules I can read, cite, version and test, on hardware I own, with client code that never leaves the laptop. No API key, no network call, nothing phoning home. On a machine used for client work that isn't a feature, it's a precondition, which is also why telemetry records rule ids, counts, model names and timings, and not question or answer text unless you explicitly turn it on. A telemetry file is the last place client code should quietly accumulate.
Same technology, different question. One asks how capable this can be. The other asks how wrong it can be without me finding out.
Verification cannot be delegated to fluency
Every mature engineering discipline eventually learns the same lesson: a system can't be trusted merely because it functions. It has to be observable, auditable, and — this is the part we keep skipping with generated code — falsifiable. We'd never run production without logs, traces and deployment history, because once causality goes opaque, reliability becomes impossible. Yet we routinely accept generated code on terms we'd never accept from a library: no provenance, no stated assumptions, no way to ask what it was checked against.
Observability can't be bolted on after a system fails. Verifiability follows the same rule. If a tool isn't designed to make its own answers checkable from the start, no amount of downstream review will conjure that property later — you'll just be reading fluent paragraphs faster.
So if one idea survives this whole article, let it be this:
The true measure of a coding assistant is not how often it is right. It's how reliably you find out when it is wrong.
Correctness can't be prompted into existence. It can't be inferred from formatting, borrowed from a real-looking citation, or assumed because the prose was confident and the rule id resolved. Someone still has to write the rule down in a form that can be checked, decide what a failing check means, and accept what the gaps cost. Those decisions stay irreducibly human — which is precisely what I want from a tool I let anywhere near a client's codebase.
Nexa is source-available under PolyForm Noncommercial 1.0.0 — free to read, run, modify and evaluate for any noncommercial purpose. It's a few hundred kilobytes of Python (stdlib only) that orchestrates open models you pull from Ollama; no weights ship with it. 32 rules today across Odoo, TypeScript and JavaScript, 27 with an automated check, 0 cross-hits.
The ARCHITECTURE.md records every decision with the evidence behind it, including the ones that were wrong first — and an honest account of what it cannot do. If you only read one file, read the limitations section.
Top comments (0)