TL;DR — AI coding assistants are gated less by context window or model capability than by whether your codebase's invariants are legible — encoded in types, tests, or linters. Where invariants are machine-checkable, assistants are genuinely excellent. Where invariants live only in a senior engineer's head, assistants will confidently produce code that compiles, passes the obvious tests, and quietly breaks something real. The fix isn't a better model; it's making your invariants explicit before you turn an assistant loose on them.
Every take on AI coding assistants eventually collapses into some version of "great at boilerplate, bad at hard stuff." That framing is comfortable and almost useless. It doesn't tell you which hard stuff, and it doesn't explain why an assistant will nail a gnarly recursive algorithm on Monday and quietly break your billing system on Tuesday with a change that looked completely reasonable.
The actual dividing line has nothing to do with algorithmic difficulty. It's about legibility: whether the invariant a piece of code depends on is written down somewhere a model — or a compiler, or a linter, or a test runner — can actually see it. Assistants aren't bounded by cleverness. They're bounded by whether the rules of your system are visible to anything other than a human's memory.
The unit of optimization is the diff, not the invariant
Coding assistants are trained and evaluated overwhelmingly on local correctness: does this function do what the docstring says, does this snippet compile, does this patch make the failing test pass. That's the diff as unit of work. It's a reasonable training signal because it's checkable at scale.
But almost nothing that matters in a real system is fully specified by the diff. A schema migration is correct only if every downstream consumer of that field is updated in lockstep. A caching layer is correct only if invalidation happens on every write path, including the one added eight months ago in a different service. A lock acquisition order is correct only if it matches every other place in the codebase that acquires the same two locks. None of these constraints live in the file the assistant is editing. They live in the relationship between that file and the rest of the system — and that relationship is exactly what a diff-shaped edit doesn't see.
Where assistants genuinely help: legible invariants
When an invariant is encoded as something machine-checkable, assistants become dramatically more useful, not because the model got smarter but because the invariant became visible. A sum type with exhaustiveness checking means an assistant that adds a new variant will get a compiler error everywhere a match statement needs updating — and it will fix those call sites correctly, because the constraint is now legible to it in the same way it's legible to a human running the build.
Same story with strong typing at API boundaries, property-based tests, contract tests between services, and linter rules that encode team conventions. In all these cases the invariant has been translated out of tribal knowledge and into an artifact the tooling can consume. Assistants are excellent at working within artifacts. Give them a type error, a failing test, a lint violation — and they will often produce a fix faster and more reliably than the human who wrote the original code.
This is also why assistants look so good in demos on statically typed, well-tested codebases and comparatively shaky in dynamically typed ones with sparse test coverage. It's not that the underlying model reasons differently about Python versus a strongly typed language. It's that the strongly typed, well-tested codebase has already done the work of making its invariants legible. The assistant is riding on infrastructure the team built, whether or not the team thinks of it that way.
Where they fail: invariants that only exist in someone's head
The failure mode isn't the assistant writing bad code. It's the assistant writing code that is locally impeccable and globally wrong, with total confidence, because nothing told it there was a constraint to violate. A few recurring shapes:
Temporal coupling. "Call
initialize()beforeprocess(), but only on the first request after a config reload." Nowhere in the type signature. Nowhere in a comment that's been updated since the reload logic was added. The assistant will happily reorder or inline these calls because nothing tells it not to.Implicit contracts across services. A field is technically optional in the schema but has been treated as required by every consumer for two years because of a business rule that exists in a design doc nobody links to anymore. An assistant asked to "clean up" the model will drop a default and break three services it never saw.
Performance-shaped invariants. A loop looks inefficient and gets "optimized" by an assistant into something asymptotically better — that also changes iteration order in a way some downstream consumer silently depended on for determinism.
Undocumented invariants baked into test fixtures. Tests pass not because the logic is right but because the fixture data happens to avoid the edge case that would expose the bug. The assistant, working from the tests as ground truth, has no signal that the fixture itself is the bug.
In every one of these cases the assistant's output is defensible read in isolation. That's what makes this failure mode expensive: it doesn't look like a bug. It looks like a plausible, well-formatted change that a reviewer skims and approves because nothing about it raises a flag — the flag would have to come from context that was never written down in the first place.
The tell: confidence is highest exactly where legibility is lowest
The uncomfortable pattern is that assistant confidence and invariant legibility are inversely correlated in exactly the situations that matter. On a well-typed, well-tested module, the assistant is appropriately confident because the guardrails will catch it if it's wrong — and it usually isn't. On a legacy module with implicit contracts and thin test coverage, the assistant is often just as confident, because nothing in its input signals "here be invariants." Confidence isn't tracking correctness here. It's tracking the absence of visible constraints, which is precisely the condition under which invisible constraints get violated.
This is a different failure than hallucination in the generative sense. The assistant isn't inventing an API that doesn't exist. It's operating correctly within a model of the system that's missing a load-bearing wall, because that wall was never drawn on the blueprint it was given.
What actually helps: invariant excavation before automation
The practical move isn't waiting for a more capable model. It's treating "make the invariant legible" as prerequisite work before delegating a change to an assistant, especially for refactors and migrations that cross module boundaries.
Concretely: before asking an assistant to touch a legacy component, write the invariant down as a type constraint, an exhaustiveness check, or a test that would fail if the invariant were violated — even a rough one. This is often faster than it sounds, because you probably already know the invariant; you've just never had to spell it out for a compiler before. Once it's spelled out, the assistant's output quality on that exact problem tends to jump, not because the model changed but because you handed it a guardrail it can actually see.
Treat undocumented invariants the way you'd treat any other form of technical debt: as a liability that compounds specifically in proportion to how much automated editing you're doing. Teams leaning hard into AI-assisted development without investing in type coverage, contract tests, and explicit invariants aren't getting a productivity multiplier. They're getting a violation multiplier, and it will show up in production, not in code review, because code review is exactly the check that a plausible-looking diff sails through.
The honest way to evaluate a coding assistant isn't "how good is the model." It's "how much of what my system depends on have I actually written down." For most codebases, the answer is less than the team assumes — and that gap, not model capability, is the real ceiling on what these tools can safely do.
Top comments (1)
The inverse correlation between confidence and legibility is the sharpest observation here. "Confidence tracks the absence of visible constraints" — that is the same failure shape as a verification gate that clears on a keyword rather than evidence: the check passes not because the property holds, but because nothing told the checker the property existed. Your "invariant excavation" framing is the right prerequisite. It is also the part most teams skip, because writing the invariant down feels like overhead until the first time an assistant confidently violates one that was never spelled out. The four failure shapes you name — temporal coupling, implicit contracts, performance-shaped invariants, fixture-baked assumptions — are all cases where the diff is locally impeccable and the system-level constraint is invisible. That is exactly the condition under which review becomes a rubber stamp.