The code runs. That's not the question.
There's a failure mode with AI-generated code that's harder to catch than bugs, security holes, ...
For further actions, you may consider blocking this person and/or reporting abuse
Yip... And that's the fatal flaw, a unit test that pass, doesnt represent a unit test in production, if it's not explicitly written to test all the edge-cases, which generally you dont do, because a 600 LOC unit test for a single function is bloat, do that for an entire codebase and you have more test code than production code, it's just not feasible. Had a case in the early days of developing V.A.L.I.D. where it would create a unit test that passes, I run the app, errors everywhere. It had written the tests based on the boilerplate it generated, instead of the variables I gave it, so it was a self-gratifying system that just made pointless tests. It also pushed for the need of constraints and the fuzzer, so it has a range to test in and the fuzzer to test whether it breaks in production. Now it's rock solid, I even built an entire autonomous accounting suite on it and saved having to write the 400k LOC that it generated for me (82% generated to be exact from a 500k LOC codebase), which is exactly where it should be, write all the unique stuff, generate all the things where mistakes could happen, but properly guarded, so they dont surface in production and before anything is shipped, run the fuzzer to test every single UI element and make sure that the rules and relations dont break.
The "self-gratifying system" description is the most honest way I've heard anyone describe AI-generated tests. That's exactly the failure mode: the AI tests its own assumptions, the tests pass, and you have a green dashboard that proves nothing about production. Your fix of adding constraints and fuzzing is the right architecture because it introduces an adversarial element the AI can't game. The AI generates the code, but the fuzzer doesn't care what the AI intended, it just tries to break things. That separation between "code author" and "code breaker" is what most teams are missing. The 82% generated ratio is interesting too because at that volume you literally cannot review everything manually, which is why the automated guardrails aren't optional at that scale, they're the only thing between you and shipping 330k lines of unverified code.
Exactly, Coca-Cola doesnt have to verify each bottle manufactured, they do spot-checks and they make damn sure their production line is flawless, with that mentality, you can produce 400k LOC of generated code, because you trust your Roslyn generator and instead of testing 1-10 bottles, you run the fuzzer to blast it for a minute at 3600 multi-dependency mutations a second. That way you know without a doubt that it's unbreakable, because it ran a year's usage in a minute. Nobody really cares that it can run that fast, for them it just means it takes less than a ms for their screen to update the total when they enter a value, but the reason it works that fast, is what makes the architecture production ready.
The Coca-Cola analogy is solid. Trust the production line, verify the output statistically rather than individually. At 400k generated lines that's really the only model that works because inspecting every bottle is impossible at that volume. The speed being a side effect of the architecture rather than the goal is a good insight too, you didn't optimize for fast, you optimized for correct, and fast came for free because the architecture had no waste in it.
Exactly, if you can be certain it will do it right 100 times, you know it can do it perfectly for 1m times. So optimize once, test twice and use forever
That's a good engineering principle in one sentence. Solid thread.
The 'functional-but-wrong code is silent' line hit hard. We spent months chasing a 97.2% coverage report that looked beautiful — until someone asked if it was testing the right things. Turns out running code ≠ working system.
The 97.2% number is a perfect example because it creates exactly the kind of false confidence this post is about. Nobody questions a test suite with 97% coverage. But coverage answers "did this code execute" not "did this code do the right thing," and those are completely different questions. You can cover every line and still test the AI's interpretation of the requirement rather than the actual requirement. That's the trap: the metrics look great precisely because they're measuring the wrong thing, and the better they look the less likely anyone is to dig deeper
Funny how 97.2% keeps finding its way into these conversations 😂 But you're right — the green bars look their best when they're measuring the wrong thing. Makes me wonder if there's a way to track "did it do what we actually asked" vs "did it run"
Ha the 97.2% is becoming the unofficial mascot of this conversation. On tracking the difference, the closest thing I've seen work is specifying acceptance criteria in terms of business outcomes before generation, not after. Instead of "does this function return the right value" you write "given a wholesale customer with a legacy discount, the final price should be X." The first one tells you if the code ran, the second tells you if it did the right thing. The gap between those two is exactly where the silent failures live, and most test suites only cover the first one because that's what's easy to automate.
Haha tell me about it — 97.2% is basically the unofficial mascot of this whole account now 😂
The acceptance criteria point is solid. Biggest thing is: a lot of teams don't skip writing business outcomes because they're lazy — they skip it because they genuinely don't know how to turn 'user should feel supported' into a test assertion. The whole thing falls apart before it even starts.
Do you find it's a training thing or a process thing when you see teams miss that gap?
Also, the emojis at the end cracked me up 🤣
Both but mostly a language problem. Engineers think in code and product people think in feelings, and nobody in the middle translates "user should feel supported" into something testable. The teams where this works have someone, usually a senior dev or a pragmatic PM, who can take a vague business outcome and break it into concrete observable behaviors. "User should feel supported" becomes "support request gets a response within 2 hours, response includes at least one actionable next step, user doesn't reopen the same ticket within 7 days." Now you have three things you can actually measure. That translation skill is rare and it's not taught in bootcamps or CS programs, people pick it up from years of watching vague requirements turn into production incidents. Process helps because you can mandate the translation step, but without at least one person on the team who's good at it the process just produces bad acceptance criteria instead of no acceptance criteria.
Your post highlights the inconvenient truth that the old and new main culprits are more often than not sloppy or nonexistent requirements. The road to hell is paved with good intentions - and implicit assumptions. Humans and machines have one thing in common: they are not mindreaders.
In your example, "search function" is not a requirement; it is a feature that needs a requirement specification. Software developers have a long tradition of filling in the requirement gaps, but it has always been a double-edged sword in my view. Developers tend to think they always know what a customer really needs, but that does not mean the customer will always agree. It does not even mean a customer is always grateful for the feedback. You could also be perceived as a troublemaker. I know what I am talking about 😉
So I think it is actually a good thing that AI takes requirements literally, because not only does it give you incredible steering power, but it also highlights the gaps in the requirements.
The "AI taking requirements literally is actually a good thing" reframe is interesting and I think you're partially right. It does surface gaps faster than a human developer who silently fills them in with assumptions. The problem is that a human filling in gaps with wrong assumptions at least produces a conversation when the result doesn't match expectations. AI filling in gaps with literal interpretation produces code that looks finished, so nobody realizes there was a gap at all until production exposes it. The troublemaker point is real though and I've lived it too. The developer who says "this requirement doesn't make sense" is doing the most valuable work on the team and is often the least appreciated for it. Maybe the best use of AI in this context is as the diplomatic troublemaker: "here's what I built from your spec, and here are the five questions your spec didn't answer that I had to guess on." That would surface the gaps without the politics.
the silent-wrong framing is right, and the part i'd add is that the green checkmark has the same failure mode as the code it's checking. a test passing tells you the code does what the test says, not what the requirement says, and those drift apart exactly the way the code and the intent do. so you can end up trusting a number (coverage, pass rate) for the same reason you trusted the clean-looking code: it looks like correctness without being tied to it.
i hit the inverted version of this today. a defect tracker said two things were broken; the tests said they'd pass; the truth was in the code, which had been fixed weeks earlier and the tracker just never caught up. functional-but-wrong and broken-but-actually-fine are the same disease, a status signal that was never re-derived against the real thing it's supposed to represent.
which is why the "verify it solves the right problem before you verify it solves it correctly" line is the whole game. the only check that doesn't drift is the one that goes back to the requirement itself. everything downstream of that, tests, dashboards, reviewer scores, is a proxy, and proxies rot quietly. how do you keep the requirement itself from being the thing that's stale? that's the layer under yours i keep running into.
The "broken-but-actually-fine" inversion is a great catch and you're right that it's the same disease. A status signal detached from the thing it claims to represent. Proxies rot quietly is a good way to put it.
On your question about stale requirements, honestly nobody has fully solved this but the least bad approach I've seen is tying requirements to observable behavior rather than documentation. A written requirement goes stale the moment someone changes the system and doesn't update the doc. But a requirement expressed as "given this input, this specific business outcome should happen" can be re-validated against the running system at any time. The requirement becomes executable rather than descriptive. It still needs a human to define it correctly in the first place but at least staleness becomes detectable because the check fails when the system drifts, instead of the doc sitting there looking accurate while the system quietly diverged six months ago. The layer under that, making sure the executable requirement itself still reflects what the business actually needs, is a human conversation and I don't think there's an automated answer for it. Someone has to periodically ask "do we still want this to be true" and that's a process discipline, not a tooling problem.
executable-not-descriptive is the right answer, and it's the same move one level down: an executable requirement is a proxy that re-derives against the running system instead of sitting static, which is exactly why it doesn't rot the way a doc does. it's tied to the thing it represents. the doc detached the moment someone changed the system; the executable check can't, because drift makes it fail.
and your last point is the honest floor: even the executable requirement is a proxy for intent, and intent is the one layer nothing automated reaches. "do we still want this to be true" can't be checked against the system because it's a question about what the system should be, not what it is. so the chain bottoms out in a human asking that on a schedule. which is maybe the real takeaway: you can push the rot further and further down by making each layer re-derivable, but the bottom layer is always someone deciding it still matters. good thread, this one clarified a lot for me.
You just described the entire proxy chain more clearly than the post did. Every layer can be made re-derivable against the layer below it, except the bottom one, which is a human deciding it still matters. That's the constraint and there's no engineering around it, only discipline through it. Good thread, learned from this one too.
that's the whole thing: re-derive every layer you can, and respect that the bottom one is a choice, not a check. discipline through it, not around it, well put. good thread, this one sharpened it for me too. catch you on the next piece.
See you there.
What I've eventually learned is to force the AI to build within its own guardrails.
Not just unit tests or prompt-level guidelines, but deterministic structural checks that make entire classes of mistakes impossible. Think of them as hooks that continuously verify architectural and coding constraints.
It's hard to produce complex example, which I certainly have, but there are some simple examples:
Import from an internal/private module -> fail
Use a deprecated API -> fail
Introduce a circular dependency or a "quick fix" local import -> fail
Violate layering or architectural boundaries -> fail
At that point, the agent stops behaving like a clever burglar trying to satisfy the prompt at any cost and starts behaving more like an engineer working under a supervision.
Once those guardrails exist, you combine deterministic with agentic check - because there are checks you can't strictly formalize, but can make it increasingly harder to fail, because the result is not supported by the test metrics.
The downside is that these guardrails aren't going to write itself. You only discover what needs to be enforced after watching agents solve real tasks for a while. Every recurring "clever monkey fix" eventually becomes another deterministic rule, linter, static analysis check, or architectural test.
In my experience, that's the real feedback loop of agentic development: observe recurring failure modes, formalize them, automate their detection. Over time, the space in which the AI can silently solve the wrong problem becomes smaller.
The "observe recurring failure modes, formalize them, automate their detection" loop is the right development process for guardrails and I think most teams skip it because they try to design the rules upfront instead of letting them emerge from actual agent behavior. Your point about the clever burglar is accurate too, without constraints the AI will find whatever path satisfies the prompt with the least effort, and that path is almost never the architecturally sound one. The layering violation check is a good example because that's one a human developer would catch through experience ("we don't import from that module") but the AI has no concept of unless you make it a hard fail. The honest tradeoff you named at the end is the part most people skip over: the guardrails cost real engineering time to build and maintain, and you can only build them after you've watched the agent fail enough times to know what to prevent. There's no shortcut for that
But this is where AI also can help.
Building proper guardrails often starts with understanding the structure of the codebase: imports, dependencies, call graphs, layering, ownership boundaries, public vs. private APIs, and so on. Once you go deeper, you quickly end up in ASTs, formal analysis, compiler tooling, and other areas that are proper CS topics.
But that does not mean you have to build the whole thing alone.
First, underlying tools for the code formal analysis already exist - Flake8 for Python, Clang libTooling with AST Matchers for C++, ErrorProne for Java. Use it as a starting point, but the whole team have to agree on the workflow first.
Then use AI to help design the fence it will later have to obey. Ask it to analyze recurring failure patterns, suggest static checks, generate AST-based rules, write custom linters, build dependency-graph validators, or explain which parser/compiler APIs are appropriate for your language.
Using the AI to help build its own constraints is a good practical shortcut. It's faster at generating AST rules and custom linter configs than writing them from scratch, and it's one of the few cases where the AI's lack of ego works in your favor since it won't push back on building a rule that restricts its own behavior. The key part of your answer is "the whole team has to agree on the workflow first" because the tooling layer is the easy part. The hard part is getting developers to commit to treating guardrail failures as hard stops rather than suggestions they override when they're in a hurry. Best tools in the world don't help if the team has a culture of skipping the check when it's inconvenient.
Yes! EXACTLY!
Imperative can be correct too, and more efficient.
Also, overuse of Monads or ML-style sideffects in functional simply make it look functional, but it becomes imperative in reality.
Yes. Good post.
Devs are very good at producing code that looks structurally right. They're also very good at producing code that confidently solves a problem you don't actually have. The gap between those two things is where Quality Assurance lives. Devs barely understand what they are doing, so you need huge teams of testers to babysit each and every little ticket.
QAs are very good at producing tests that look structurally right. They're also very good at producing tests that confidently cover an infinite graph you don't actually have. The gap between those two things is where Mathematics lives. QAs barely understand what they are doing, so you need huge teams of theorists to babysit each and every little QA Institution and training center.
Mathematiciaians are very good at producing proofs that look structurally right. They're also very good at producing conundrums you don't actually have. The gap between those two things is where tentacle-armed unimaginable multidimensional space horrors live. Mathematicians barely understand what they are doing, so you need huge teams of tentacle-armed unimaginable multidimensional space horrors to babysit each and every little planckian frame of this Big Brother show called Universe, which we are streaming now in HD Dimension to our customers.
Ha the turtles-all-the-way-down escalation from devs to QAs to mathematicians to cosmic tentacle horrors is a journey I did not expect this comment section to take. To clarify though, the "functional" in the title means "the code functions, it runs, it works" not functional-vs-imperative as a paradigm. The point is that code can execute correctly and still solve the wrong problem, regardless of whether you wrote it in Haskell or assembly. But I appreciate the interdimensional QA framework, will update our review checklist accordingly.
That part of my comment was a joke.
I use jokes for testing System 1 vs. System 2 capabilities of both the carbon and the silicon based workforce.
All other parts of my comment were not jokes.
They were a rebuttal of your post's naively optimistic main conclusion.
If we cannot vouch for our own code, as the large corpus of CVEs and meme level left-pad facepalm blunders show, then what chance do we have if throughput is increased?
One can argue pro/con for llms' capabilities when it comes to code generation.
For me, that's a moot point.
My argument is that we humans were outgunned by sheer volume of 'ready for testing' tickets since the 1960s.
Due to broculture, rampant egoism and financial interests, this was not openly acknowledged by the programming and corporate community.
We are still in the denial phase. LLMs simply made the symptom more pronounced, but they did not cause it.
To my surprise, we are still in denial phase.
But more importantly as I showed: We even lost our humor. Which is sad.
If life seems jolly rotten
There's something you've forgotten
And that's to laugh and smile and dance and sing
The 'functional but wrong' failure mode is the one that should scare teams most, because it passes every gate we built to catch the loud failures. Broken code announces itself; code that confidently solves the wrong problem ships clean and lives for months. Your point about the friction of writing being where misunderstandings surface is key, the agent skips the part where a human would have flagged that the requirement itself was wrong. Tests prove the code runs. Nothing in the pipeline proves it does the right thing. That gap is where the next generation of checks has to live, verifying intent, not just behavior.
"Verifying intent, not just behavior" is a clean way to frame where the tooling gap is right now. The entire CI/CD pipeline was built to answer "does this work" and nobody built the equivalent pipeline for "is this right." That's the missing layer and I don't think it's a tooling problem, it's a process problem, because intent lives in humans not in code. The closest proxy is concrete acceptance criteria written before generation, but even that only catches the intent that someone thought to write down.
This is an important distinction that doesn't get discussed enough. AI-generated code can be syntactically correct, pass tests, and still miss the actual business intent. We've found that the biggest wins come from validating requirements and edge cases before focusing on implementation quality. It's a pattern we see often while building production AI applications at IT Path Solutions. The engineering challenge isn't just generating code it's making sure it solves the right problem.
Agreed. Requirements validation before implementation is where most teams underinvest, especially when AI makes the implementation step feel instant. Thanks for reading.
Really enjoyed this. I think the solution isn't just better prompting, it's better delegation. Give AI a complete task with clear success criteria, then review the outcome instead of every line of code.
I recently shared my thoughts on this in post in my blog post => Asking vs Delegating AI Agents 🧐
The delegation framing makes sense. Clear success criteria upfront changes the review from "read every line" to "did it meet the spec," which scales way better when the volume of generated code is high. Thanks for reading.
The silent-wrong failure mode is the one I trust least, precisely because every automated check stays green. The friction you describe, realizing mid-build that the spec itself is wrong, is the part AI removes, and that friction was doing real validation work. I've started treating "passes tests" as evidence the code does something, never evidence it does the right thing.
"Passes tests as evidence the code does something, not evidence it does the right thing" is a clean reframe and I'm going to steal that for future conversations. The friction point is the part I keep coming back to as well. Nobody designed that friction on purpose, it was just a side effect of building being slow, and we only realized it was doing validation work after we removed it.
This is why requirements matter more than code generation. AI can write functional code, but only humans can validate business intent, edge cases, and real user needs. Correct logic starts with correct understanding.
Correct understanding before correct implementation. That's the whole sequence and AI didn't change it, just made it easier to skip.
The "self-gratifying test" problem is real, AI generates code, then generates tests that validate its own interpretation of the requirement, not the actual requirement. Green dashboard, wrong system.
The only fix I've found is writing the acceptance criteria before touching the AI at all. Not prompts, actual business scenarios with real numbers. If you write the test first, at least the AI is generating code to pass your definition of correct, not its own.
That's exactly the sequence. Acceptance criteria first, generation second, validation third. The teams that reverse the order end up with the self-gratifying loop where the AI defines what correct means and then proves itself right. Your point about real numbers matters too because vague criteria like "should handle discounts correctly" gives the AI room to interpret. "Wholesale customer with 12% legacy discount on a $450 order should pay $396" gives it nowhere to hide. The more specific the acceptance criteria, the less room for the silent-wrong failure mode.
The gap you are naming is real, and the fix has the same trap one level up. Going back to the requirement instead of the prompt only helps if the requirement is held by someone other than the person who wrote the prompt. If the prompt was wrong because the author's model of the requirement was wrong, the same author re-reading the requirement carries the same blind spot and signs off. The friction you credit in hand-written code was never just the act of building, it was the requirement pushing back through an independent party: the client who meant fuzzy match, the teammate who knew the grandfather clause, the edge case that filed a complaint. txdesk named the load-bearing part in the comments, the test shares an author with the code. So does your re-read with your prompt. Validation catches functional-but-wrong only when the validator holds context the generator never had. Re-reading your own spec at full confidence just confirms the misunderstanding in cleaner words.
That's a legitimate hole in the argument and I should have been more explicit about it. The "go back to the requirement" step only works if someone other than the prompt author does it, because the blind spot that produced the wrong prompt will happily confirm the wrong output. You're describing the same single-author problem that makes self-review useless in traditional code review, just moved up a level. The friction in hand-written code worked because the codebase itself pushed back, the compiler complained, the test failed, a teammate asked "why did you do it this way." All of those were independent of the author's mental model. The practical fix is the same as it always was: a second person with different context. The author validates that the code matches their intent, someone else validates that the intent matches the actual need. That second check is the one most teams skip because it's slower and harder to staff for, but it's the only one that catches the blind spot you're describing.
Agreed, and the part I'd nail down is that the independence has to be in the context, not the headcount. A second reviewer handed the same ticket inherits the same wrong framing and signs off on it, because you've added a person but not a source of truth. Two people reading one spec is one mental model with a backup copy.
The check only bites when the second person is bound to something the author doesn't control: the actual user, the real requirement, the failing case from production. That's why the compiler and the failed test worked in your hand-written example, they answered to the machine, not to your story about the machine. The fix isn't 'get a second person,' it's 'get a second source of truth and put a person in front of it.'
"A second source of truth, not a second person" is a better formulation than what I had. Adding a reviewer who reads the same ticket is just double-checking the same wrong map. Adding a reviewer who talks to the actual user or looks at production behavior is checking the map against the territory. The compiler worked as validation not because it was smart but because it was independent, it answered to the machine's rules not the developer's assumptions. The equivalent for business logic is the actual user doing the actual thing, which is harder to put in a pipeline than a compiler but it's the only oracle that can't be fooled by a well-written spec that describes the wrong behavior.
The "functional but wrong" problem is why I don't trust demo videos anymore. The code looks right, the types check out, the tests pass -- and it's solving the wrong problem.
The root cause is that AI optimizes for the prompt, not the requirement. Those are different things. The prompt is what you typed; the requirement is what the user actually needs. Humans bridge that gap through friction -- you read the spec, you question it, you realize halfway through that the ask doesn't make sense. AI skips that entire step.
I've started writing contracts before the code, AI-generated or not. Not formal specs, just concrete examples of inputs and expected outputs. When the code passes but the examples don't, that's your signal that it's functional but wrong.
How do you catch this before production? Code review catches bugs, but "wrong problem" is harder to spot in a PR.
The contracts-before-code approach is the right fix and it's basically the same conclusion several people in this thread have landed on independently: define what correct looks like before you generate anything. The "wrong problem" detection in PR review is genuinely hard because the reviewer is usually reading the same ticket as the author, so they carry the same assumptions. The only version of review that catches it is when the reviewer checks the code against the concrete examples you wrote, not against the ticket description. That's why your input/output contracts work, they give the reviewer something independent to validate against instead of just reading the code and nodding because it looks reasonable. Without those examples, "wrong problem" is basically invisible in a diff.
The “functional but wrong” failure mode is exactly where engineering judgment still matters. Tests can confirm behavior against a prompt, but they do not automatically confirm that the prompt captured the real business rule. I’ve seen this become especially risky when AI-generated tests validate AI-generated assumptions, creating a green dashboard that proves very little. The stronger pattern is to separate generation from validation: domain constraints, adversarial cases, fuzzing, and human review should challenge the interpretation, not just the implementation.
The "AI-generated tests validating AI-generated assumptions" loop is the core trap and you described it well. Green dashboard that proves nothing. Separating generation from validation is the fix, and the key word in your list is "adversarial." The check only works when it's trying to break the code's interpretation, not confirm it.
The "business rules get flattened" section hits hard. I watched this happen with a pricing module last quarter. The AI implemented the discount logic exactly as the ticket described. What the ticket did not mention was the legacy tier system from the 2023 acquisition that gave certain accounts a locked-in rate. A human developer would have known to ask about edge cases. The AI just shipped it.
The validation gap you describe is real, but I think there is a deeper issue. The friction of writing code manually is not just slowdown. It is a forcing function for understanding. When you have to type out every line, you confront the requirement gaps as you go. AI removes that friction, and with it removes the discovery process.
The teams I have seen handle this well treat AI output as a first draft, not a finished PR. They do the validation work you mention, but they also add a step: someone reads the code line by line and explains what it does back to the team. Sounds slow, but it catches the "functional but wrong" problem faster than any test suite.
Have you found any validation patterns that work specifically for business logic, as opposed to pure function correctness?
The pricing module story is a textbook case. The AI did exactly what the ticket said and the ticket was incomplete. Nobody's fault and everybody's problem.
On validation patterns for business logic specifically, the one that's worked best for us is what I'd call "example-driven specification." Before any code gets generated, someone writes three to five concrete scenarios with real numbers that cover the normal case, the weird case, and the case nobody wants to talk about. For your pricing module that would look like: "standard customer ordering 100 units pays X, legacy tier customer from the 2023 acquisition ordering 100 units pays Y, legacy customer whose contract expired last month pays Z." Those become the acceptance tests. The AI can generate whatever it wants but it has to pass those specific scenarios, and writing them forces the team to surface the edge cases before generation rather than discovering them in production.
The "explain it back to the team" step you mentioned is underrated and it catches a different failure than testing does. Tests catch wrong behavior. Explaining catches wrong understanding. Both matter but the second one is harder to systematize which is probably why most teams skip it. The read-and-explain step is basically a manual forcing function to replace the understanding that used to come for free when you wrote the code by hand.
Part of what makes 'functional but wrong' code so insidious is that it often looks better than correct code — it's cleaner, better formatted, and passes every linter. The messy parts (edge cases, business logic exceptions, domain-specific constraints) are exactly what LLMs smooth over because they generate the most probable path, not the most correct one.
The practice that's helped me most is writing acceptance criteria as concrete business scenarios before generating any code. Not user stories — specific examples with real numbers and edge cases. A wholesale customer with a 5-year legacy contract ordering 300 units during a promo blackout period should pay X. If the generated code can't pass that, it doesn't matter that it passes every unit test.
It shifts the validation from 'does the code run?' to 'does the system solve the actual problem?' — which is exactly where engineering judgment lives.
"Most probable path, not most correct one" is a precise description of what LLMs actually optimize for. That framing explains why the output looks cleaner than correct code, because correct code has the ugly real-world exceptions baked in and those aren't probable patterns in training data, they're specific to your business. The wholesale example with the promo blackout period is exactly the level of specificity that makes acceptance criteria useful. Vague criteria give the AI room to be confidently wrong. Concrete numbers with real business constraints give it nowhere to hide.
"This is exactly why we built AINAScan — we found that AI-generated code passes all tests and 'works', but consistently produces the same 15 structural bugs: save functions that never write to DB, async functions with no await, parameters that have zero effect on return values. Linters miss all of these. The code looks fine until production."
The "parameters that have zero effect on return values" one is a good example of something that's syntactically valid but logically dead. A linter sees a function with parameters and a return value and moves on. It doesn't check whether the parameters actually influence the output. That category of structural bugs is interesting because they're technically not bugs in the traditional sense, the code runs fine, it just doesn't do what the signature promises it does.
I completely agree.
One thing I've noticed while building AurX is that AI often produces code that works, but that doesn't automatically make it a good solution.
A function can return the expected result while still introducing hidden complexity, poor architecture, security issues, or performance problems.
I've started treating AI like a fast collaborator rather than an authority. It accelerates development, but the responsibility for understanding, reviewing, and maintaining the code still belongs to the developer.
In my opinion, the most valuable skill in the AI era isn't generating code—it's being able to recognize when generated code is actually the right solution.
"Fast collaborator rather than an authority" is the right mental model. The teams that get into trouble are the ones that treat AI output as a suggestion that was already vetted instead of a first draft that needs judgment applied to it. Your point about recognizing when generated code is the right solution is the underrated half of the skill. Everyone talks about prompting better. Fewer people talk about evaluating better. And evaluating is the harder skill because it requires understanding the problem deeply enough to spot when a clean looking solution is actually wrong for your context.
This is precisely what I'm working on with ReWeaver AI. We are both speaking the same language on this topic. I'm actually giving a talk at the AI Engineer World's Fair next week called "The Design-Code Roundtrip That Isn't." Such an important discussion to be having now.
Good talk title, the "roundtrip that isn't" framing captures the gap well. Hope the talk lands, it's a conversation that needs more concrete examples right now.
The tell I'd add: "functional but wrong" usually shows up in a diff as code that's suspiciously clean.
Real domain logic is lumpy. It has the guard clause for the legacy accounts, the weird branch for the maintenance window, the comment that says "don't touch, this is for the 2023 migration." Those ugly parts aren't noise — they're scar tissue from every time the happy path met reality. When a generated function comes back smooth with none of that, the reflex is "nice, clean code." But the domain didn't get simpler; the exceptions just got averaged away, because your grandfather clause isn't on the probable path.
So the review habit that's saved me is to stop asking "does this look right" and start asking "where are the ugly parts?" If a function that touches pricing or permissions has zero special cases, that's not elegance — that's a missing requirement in a clean shirt. The absence of friction is the signal.
AI-generated code can be functionally correct but still fail to meet the actual business requirement. The real engineering challenge is validating intent, business rules, and edge cases not just whether the code compiles or passes tests.
That's the whole post in three sentences. Compiles and passes tests is the floor, not the finish line.
Correct
Thanks for reading.