One incident still sticks with me. Our application services have to update a batch of data in shared memory atomically. That memory is managed centrally by the system: cross-process locks, memory ordering, and version switching are all wrapped inside the .so library, and callers reach it through a single shared header.
At the time, though, the library offered no interface for "atomic bulk update". The AI's approach was about as blunt as it gets: grab the physical address through mmap, overwrite field by field according to its own reading of the layout, and casually invent a flag on the side to serve as a "currently being modified" lock.
In a single process it ran fine, and the tests passed. The trouble is that what it bypassed was not an ordinary function but an entire system contract: the cross-process lock no longer held, the memory-ordering guarantees were gone, and any other process reading that memory could hit half-written garbage at any moment. Worse, the bypass leaves no trace. When the library later needs auditing or a memory layout change, those "ghost call sites" poking straight at physical addresses will be impossible to track down.
The AI wasn't out to sabotage anything. It simply took the path of least resistance when nothing provided the capability it needed. And that path happened to water down the contract "atomicity is guaranteed by the layer below" into "every call site makes up its own guarantee".
The longer I use it, the more of these moments I collect: locally flawless, globally full of holes. They tend to show up in three ways:
First, abstraction boundaries collapse. Like the mmap case above. In ordinary development, when a library lacks an interface, the caller and the provider sit down and agree on where the boundary is and how to define it. The AI makes no such judgement; wherever the rules don't reach, it fills in the gap itself.
Second, nothing accumulates in the foundation layer. An experienced engineer will pull common logic into a shared class along the way. AI tends to solve in place: this feature needs it, so add a method to the current file; the next feature needs it too, so write it again in another file. Nobody breaks the foundation layer — it simply never gets built.
Third, layered conventions get ignored. In any moderately complex system, technical constraints come in layers. Take the industrial control and data acquisition system we build: the platform layer is strict C++11 with exceptions disabled, while the application layer above it may use C++14. That means when creating a smart pointer, the platform layer has to write std::unique_ptr<T> p(new T(args...)) by the book, while the application layer can use std::make_unique. To the AI, though, the whole repository is flat. The gcc in the dev environment is new enough that make_unique slips into the platform layer and still compiles; the problem only surfaces at integration testing.
A person might remember that C++11 rule. What a person can't do is spend an afternoon driving the AI through a dozen-plus sessions and five features in quick succession, and still remember which cell of the layering diagram the current session is sitting in.
Pull these observations together and they point at one thing: when AI coding spans sessions, the contracts between features lose their carrier.
System software is inevitably assembled from hundreds or thousands of features. You can't opt out of contracts; you can only choose whether to write them down, or to keep them in your head and carry them in prompts. Experience shows the head doesn't retain them, and stuffing them into prompts is a long-shot bet — miss one layer of context and you have a production incident.
Someone might ask: aren't SDD (spec-driven development) and Code Plan First (plan-first mode) meant to do exactly this?
No. The reach of both patterns stops at "get the current task right". You spell out the interface rules in session 1's spec, and nothing guarantees the AI still remembers them in session 50. You can, of course, copy "must go through the library's single entry point" and "no make_unique in the platform layer" into every feature's detailed design, over and over. That is unreasonable to expect of anyone, and it will inevitably miss something. A problem that sits at the boundary of a pattern can't be patched by "using the tool better".
Traditional documentation is even less help. It gets split by role into isolated islands of requirements, architecture, and testing, in wildly varied formats — I have seen companies whose requirements docs are PowerPoint decks because that makes them easier to present upward. A few rounds of code iteration later, the documents contradict each other. You can't ask an AI to consult a pile of stale dead artifacts like that.
So the problem is not AI's comprehension or its output quality — those have actually exceeded my expectations. To put it plainly: AI is not the problem. Our way of organizing is. Our current model of project management cannot keep up with this new productivity.
To hold the contracts between features in system software, we need an independent carrier that outlives a session — one that both a person and an AI can read precisely, and whose structure makes explicit which rule governs which layer.
So along what dimension should this long-lived document be organized, if we don't want to repeat the same mistake?
Top comments (0)