Software engineering is wrestling with cognitive debt — the loss of shared understanding that accelerates when AI generates code faster than humans can assimilate it. The contradictions feel new. They aren't.
Six other domains faced the same structural contradictions, often at stakes where failure meant death. Each found a resolution. None of the resolutions involved slowing down.
1. The Velocity-Comprehension Gap
The software contradiction: AI generates code faster than humans can understand it. Speed goes up, comprehension goes down.
The extreme version: Nuclear reactor operations
A nuclear reactor's state changes faster than any operator can track. Neutron flux, coolant temperature, pressure, control rod positions — dozens of variables shifting continuously, interacting nonlinearly. The system's state space is vast and the consequences of misunderstanding are catastrophic.
Early reactor operations attempted the braking approach: slow the process down, add more operators, require more oversight. Three Mile Island (1979) proved that more operators watching more dials didn't prevent misunderstanding — it amplified it. The operators misinterpreted the reactor's state for hours because they were tracking individual variables instead of system properties.
How nuclear power solved it
The industry shifted from monitoring mechanism to monitoring the operating envelope. Operators don't track how the reactor works moment to moment. They verify that the reactor stays within a defined envelope. The bounds the plant's Technical Specifications and operating license make legally binding: temperature below X, pressure below Y, neutron flux within band Z. They call these limits Tech Specs. A software engineer would call them invariants. They are the same idea. If any limit is violated, the protection system acts automatically. The operator's job shifted from understand the reactor's state to verify the reactor satisfies its invariants.
The operating envelope is a small, stable, comprehensible artifact. The reactor dynamics are vast, fast, and incomprehensible in real time. Understanding scales with the envelope, not with the dynamics.
The carry-over
Replace reactor state with codebase state. Replace operating envelope with system invariants. The developer's job shifts from understand what the AI-generated code does to verify the code satisfies its declared properties. The properties are small, stable, and human-readable. The code is vast, fast-changing, and increasingly AI-generated. Understanding scales with the properties, not with the code.
2. Bypass of the Mental Struggle
The software contradiction: AI provides solutions instantly, so developers skip the problem-solving process that builds deep understanding.
The extreme version: Medical diagnostics
AI now outperforms radiologists at detecting certain pathologies from imaging. A medical student who relies on AI for diagnosis never develops the clinical judgment that comes from looking at thousands of scans, making mistakes, and building pattern recognition through struggle. The efficiency gain (faster, more accurate diagnosis) directly undermines the learning process (internalization through difficulty).
Medical education faced this contradiction a decade before software did. If AI diagnoses better than residents, why have residents diagnose at all?
How medicine solved it
Medical education didn't eliminate the struggle. It redirected it. The learning objective shifted from learn to produce the diagnosis to learn to evaluate whether a diagnosis is correct. Residents now learn to assess AI outputs: Is this diagnosis consistent with the patient's history? Does the confidence score match the imaging findings? What would change the diagnosis?
The struggle moved up one level of abstraction. The production skill (finding the answer) was automated. The evaluation skill (judging the answer) was preserved and deepened. Evaluation requires understanding the domain — the pathology, the patient context, the differential diagnosis — even when the answer is handed to you.
Chess followed the same trajectory. AI plays better than any grandmaster. Chess education didn't die. It shifted from learn to find the best move (search, automatable) to learn to evaluate positions (judgment, not automatable). The evaluation skill is more durable and more transferable than the search skill.
The carry-over
The developer's struggle shouldn't be "how do I implement this?" (production, now automatable). It should be "is this implementation correct for our context?" (evaluation, not automatable). Evaluation requires understanding the domain constraints, the architectural rationale, the failure modes — exactly the things cognitive debt erodes. Redirecting the struggle from production to evaluation preserves the mental model while accepting AI-speed implementation.
Developers write the specification (struggle preserved — "what must be true?") and evaluate the AI-generated implementation against it (judgment preserved — "does this satisfy what must be true?"). The specification is the externalized mental model.
3. Loss of Intentionality — The Why Gap
The software contradiction: AI generates code based on patterns, not strategic intent. The why behind decisions is never recorded because no human deliberated over the implementation.
The extreme version: Legal systems
Imagine a legal system where courts issued verdicts without written opinions. Guilty. Not guilty. No reasoning, no precedent, no explanation of which laws applied or how the evidence was weighed. Future courts would have no basis for consistent rulings. The system would be functional (verdicts would be rendered) but incoherent (nobody would know why).
This isn't hypothetical. Early legal systems worked this way — and collapsed under their own inconsistency. The resolution is one of civilization's oldest intellectual technologies.
How law solved it
Every judicial decision in common-law systems must include a written opinion — the court's reasoning, the applicable precedents, the rejected arguments, the interpretation of law applied to facts. The opinion is the law, not the verdict. Future courts apply the reasoning, not the conclusion.
The opinion is a first-class artifact. It's published, searchable, citable, and debatable. It survives the judge's retirement. It survives the court's dissolution. The why is not an oral tradition or a memory — it's a document with formal standing.
The carry-over
Architectural decisions need opinions, not just verdicts. "We chose gRPC over REST" is a verdict. "We chose gRPC over REST because the threat model assumes network-level adversaries requiring mutual TLS, and gRPC's native support for mutual TLS reduces the authentication implementation surface compared to REST's header-based token model" is an opinion.
The opinion is recorded as a constraint with rationale: the constraint captures the "what" (mutual TLS required), the rationale captures the "why" (threat model assumes network-level adversaries). Both are version-controlled. Both survive team turnover. The "why" isn't a memory — it's a file.
When AI generates code, it generates a verdict (the implementation). The opinion (the rationale) must exist before the verdict, as an input to code generation, not an afterthought. The constraint with rationale is the legal opinion equivalent in software architecture.
4. Distributed Cognitive Fragmentation
The software contradiction: Multiple AI agents touch different parts of the codebase simultaneously. The system's theory shatters. No single human holds a coherent view of how the components interact.
The extreme version: The International Space Station
The ISS was designed and built by five space agencies across 15 countries. Modules were engineered in different languages, using different standards, on different continents, over decades. No single engineer — not even a single agency — understood the entire station.
If the ISS depended on holistic understanding in any one mind, it would never have been built. The complexity exceeded human cognitive capacity before the first module launched.
How aerospace solved it
Interface Control Documents (ICDs). Every module has a formal specification of its interfaces: what it provides (power output, data format, structural load capacity), what it requires (cooling capacity, data bandwidth, mounting points), and what it guarantees (vibration tolerance, thermal range, failure modes).
The modules don't need to understand each other. They need to satisfy their interface contracts. The station is coherent because the contracts compose, not because any mind holds the whole picture.
The Internet works the same way. No one understands the entire Internet. Coherence comes from protocol specifications (RFCs). Each implementer understands their layer's contract. TCP doesn't know about HTTP. HTTP doesn't know about your application. Each layer satisfies its specification. The system composes.
The carry-over
Software systems need ICDs, not holistic understanding. When multiple AI agents modify different modules simultaneously, coherence comes from interface contracts that each modification must satisfy, not from a human who understands all the modifications.
Each module has a contract: what it provides, what it requires, what it guarantees. AI-generated changes within a module are verified against that module's contract automatically. Cross-module changes are verified against the interface contracts between the affected modules. The shared theory isn't shared understanding — it's a shared contract repository.
The shift: from "we need someone who understands the whole system" (impossible at scale, fragile, doesn't survive turnover) to "we need contracts that compose correctly" (scalable, durable, machine-verifiable).
5. Invisible Coordination Overhead
The software contradiction: More AI agents means more invisible decisions. Each invisible decision is information a human should know but doesn't, creating unknown unknowns that paralyze the team.
The extreme version: Military operations
A combined-arms military operation involves hundreds of units acting simultaneously. Each unit makes decisions every minute — where to move, what to engage, when to hold. Each decision affects adjacent units. An artillery battery firing on a position it believes is enemy-held, while friendly infantry is advancing toward the same position, produces fratricide. The decision to fire was locally rational and globally catastrophic.
The invisible-decision problem in military operations is measured in lives. The braking solution — require every unit to clear every decision with headquarters — is called centralized command. It failed catastrophically in every major conflict where it was tried, because the communication overhead exceeded the speed of events.
How the military solved it
Commander's Intent. The doctrine states: the commander communicates the desired end state, not the specific actions. "Seize the bridge by 0600 and deny enemy crossing" is the intent. Each subordinate unit chooses its own actions as long as those actions advance toward the intent.
Decisions are still invisible to headquarters. But they're bounded: every decision must satisfy the commander's intent. A unit that makes a locally rational decision that violates the intent is wrong, regardless of local circumstances. The intent is the constraint. The decisions are free within the constraint.
Financial trading follows the same pattern. Algorithmic trading systems make thousands of invisible decisions per second. The resolution isn't to make every decision visible (impossible at that speed). It's to define risk limits and circuit breakers: the trading algorithm can do whatever it wants as long as position sizes stay below X, daily loss stays below Y, and sector concentration stays below Z. Violations trigger automatic halts.
The carry-over
AI agents in a codebase need commander's intent, not centralized review. The intent is expressed as architectural constraints: "all inter-service communication uses mutual TLS," "retries use exponential backoff capped at 5 attempts," "no module may import from an upstream layer." Each constraint is the commander's intent for one architectural dimension.
AI agents make invisible decisions within those constraints. The decisions don't need to be visible because they're bounded. A decision that violates a constraint is caught by the CI gate — the software equivalent of the circuit breaker. The team doesn't need to know about every decision. They need to know the constraints are satisfied.
The shift: from "make all decisions visible" (impossible at AI speed, kills throughput) to "make all constraints explicit and verify automatically" (scalable, preserves speed, catches violations).
6. Re-Deriving What's Already Settled
The software contradiction: Every developer (and every AI agent) re-reasons routine work from scratch — how to wire a service, handle an error, deploy a change. The reasoning is redone on every task, burning cognitive budget on problems the team already solved, and producing a slightly different answer each time.
The extreme version: Aviation
Flying a modern aircraft involves hundreds of steps where a single omission is fatal — fuel state, control surfaces, hydraulics, pressurization, flap settings. A pilot cannot re-derive correct procedure under time pressure on every flight, and the cost of one forgotten step is increase in risk to human life. Early aviation tried to rely on individual skill and memory. It killed pilots at a rate the industry could not sustain.
The turning point is famous: in 1935, Boeing's Model 299 — the prototype B-17 — crashed on takeoff because the crew forgot to release a control lock. The aircraft wasn't too complex to fly. It was too complex to fly from memory.
How aviation solved it
The checklist and the Standard Operating Procedure. Pilots don't re-derive how to fly the plane. They execute a pre-validated theory — a written, tested sequence — that someone reasoned through carefully, once, under no time pressure, and that everyone now runs identically. The checklist isn't a crutch for weak pilots; it's the mechanism that lets expert pilots spend their judgment on the situation in front of them instead of on remembering the settled parts.
The cognitive load of the routine is offloaded onto an external artifact. Working memory is freed for the part that actually requires a human: the novel, the ambiguous, the off-nominal. The SOP encodes the consensus answer so it never has to be re-litigated mid-flight, and so two different crews behave the same way in the same situation.
The carry-over
AI agents are tireless re-derivers. Asked to add a service, an agent will re-reason routing, error handling, retries, and deployment from first principles every time — and arrive somewhere slightly different on each pass. That is cognitive debt generated at machine speed: a thousand locally plausible answers to questions the team already settled.
The fix is the same as the cockpit's. Encode the settled decisions as executable SOPs — scaffolds, templates, golden paths, lint rules, the constraints the previous five sections describe — so neither human nor agent re-derives them. The reasoning happens once, deliberately, and is captured in an artifact every subsequent change runs against. Human and AI judgment is then spent where it's irreplaceable: the off-nominal case the checklist doesn't cover.
The Meta-Pattern
Every domain that faced the velocity-comprehension contradiction at life-or-death stakes solved it the same way:
| Domain | Implicit theory (fragile) | Explicit artifact (durable) |
|---|---|---|
| Nuclear operations | Operator's mental model of reactor state | Operating envelope / Tech Specs (invariant set) |
| Medicine | Clinician's diagnostic intuition | Evaluation rubric + outcome metrics |
| Law | Judge's reasoning in chambers | Written opinion with precedent |
| Aerospace | Engineer's holistic system understanding | Interface Control Documents |
| Military | Commander's knowledge of all unit decisions | Commander's Intent doctrine |
| Aviation | Pilot's memory of correct procedure | Checklist / Standard Operating Procedure |
A note on measurement, because it's the trap that hides all of the above. You can't measure understanding directly, so teams measure what's easy — velocity, output, throughput — and reward speed while cognitive debt compounds invisibly. Healthcare made this mistake for decades, measuring procedures performed instead of patient outcomes, until iatrogenic harm became a leading cause of death while every dashboard stayed green. The correction was outcome metrics layered on top of throughput metrics. Software's equivalent proxies: specification coverage (what fraction of behaviors are governed by explicit constraints), constraint violation rate (how often new code diverges from the declared theory), unspecified change ratio (how much change lands in modules with no constraints at all), and constraint staleness (how many constraints predate their module's last major rework). These don't replace velocity metrics. They make the invisible cost visible next to the visible one.
Software is the last major engineering discipline still depending on implicit, in-head theory as the primary carrier of system understanding. Every other high-stakes domain externalized the theory into a formal artifact, because the stakes forced them.
AI didn't create this problem. It raised the stakes. When humans wrote all the code, the implicit theory eroded slowly — through turnover, through growth, through time. Braking mechanisms (code reviews, pair programming, documentation sprints) managed the erosion at human speed.
At AI speed, the erosion is instant. The braking mechanisms can't keep up. The implicit theory shatters on contact with AI-speed development, the same way centralized military command shatters on contact with the speed of modern warfare.
The resolution is the same one every other domain found: stop storing the theory in heads. Store it in artifacts. Make the artifacts formal, composable, version-controlled, and machine-verifiable. Then speed doesn't erode understanding, because understanding was never in the thing that speed affects.
The nuclear engineer doesn't understand the reactor's moment-to-moment dynamics. The engineer understands the operating envelope. The envelope is 50 pages. The reactor dynamics fill textbooks.
The software developer doesn't need to understand the AI-generated code's moment-to-moment logic. The developer needs to understand the system's invariants. The invariants are a few hundred lines. The code is millions.
The theory was always too important to store in memory. We just didn't know it until the speed made the fragility visible.
Top comments (0)