DEV Community: Bala Paranj

From Fallacies to Superpowers: Eight Agent Skills That Make AI-Assisted Development Work

Bala Paranj — Sun, 07 Jun 2026 11:59:32 +0000

The Fallacies of GenAI Development named eight assumptions that break AI-assisted development. The resolutions were framed as human knowledge — things the engineer must understand and apply.

But the resolutions don't have to live in the engineer's head. They can live in the agent's workflow.

Projects like Superpowers proved that agents can follow structured methodologies — brainstorm before coding, write tests before implementation, review against specs before declaring success. The skills are mandatory workflows, not suggestions. The agent checks for relevant skills before any task.

The same approach works for the Fallacies resolutions. Each one can be encoded as an agent skill that fires automatically. The engineer doesn't need to remember "check for existing libraries before generating." The agent does it as a mandatory step.

Here are the eight skills. Each one resolves one fallacy. Each one is achievable today with current agent capabilities.

But first, a critical boundary.

The line between agent skill and human judgment

Not everything in a Fallacy resolution should be automated. The Fallacies series itself warns against this — Fallacy #3 (AI can't verify AI) and Fallacy #4 (dropping review) exist because teams automated judgment calls that should have stayed with humans.

Each skill below has two halves:

The mechanical half (agent does this): Search for existing libraries. Run the compiler. Execute the linter. Read the specification file. Count the boundaries. These are deterministic actions with deterministic outputs. The agent executes them. No judgment required.

The judgment half (agent surfaces this to the human): "Is this the right library?" "Does this architectural constraint still apply?" "Should this uncovered decision become a new spec?" These require context, domain knowledge, and strategic thinking. The agent surfaces the question. The human answers it.

The agent does the LEGWORK. The human makes the CALL. The agent that tries to make the call is Fallacy #3 — using AI to verify AI. The agent that doesn't do the legwork is wasting human attention on mechanical work (Fallacy #4).

WRONG:  Agent decides "this library is the right choice" → Fallacy #3
WRONG:  Human searches for libraries manually             → Fallacy #4
RIGHT:  Agent searches, presents 3 options with tradeoffs → human picks

Every skill below respects this boundary. Watch for the split: steps marked [MECHANICAL] are what the agent does autonomously. Steps marked [SURFACE] are what the agent presents for human decision.

Skill 1: Compose-First

Resolves Fallacy #1: Faster generation ≠ faster engineering

When it fires: Before generating any implementation code.

What the agent does:

1. [MECHANICAL] Parse the task: what capability is needed?
2. [MECHANICAL] Search for existing functions in the codebase that already provide it
3. [MECHANICAL] Search for well-maintained upstream libraries that provide it
4. [SURFACE]    Present findings: "Found 2 existing options: [library A] (last updated 
                3 days ago, 12k stars) and [library B] (last updated 8 months ago, 
                200 stars). Also found internal utils/retry.go with similar logic.
                Shall I compose from one of these, or generate new?"
5. [MECHANICAL] After human picks: write the import + glue code
6. [MECHANICAL] Log the decision: "Composed from [library] per human approval"

Why this is a superpower: The agent that composes instead of generating produces 80-95% less code for the same capability. Less code = less to maintain, test, debug, secure. The agent becomes a librarian, not a typist. The codebase shrinks while capabilities grow.

What it looks like in practice:

Without skill:
    Task: "Add HTTP retry logic"
    Agent: generates 150 lines of retry implementation

With skill:
    Task: "Add HTTP retry logic"  
    Agent: "Found existing retry library in go.mod dependencies.
            Writing 6 lines of configuration instead of 150 lines
            of implementation."

Skill 2: Property-Check

Resolves Fallacy #2: Plausible ≠ correct

When it fires: After generating any code, before presenting to the human.

What the agent does:

1. [MECHANICAL] Read the project's property definitions (if they exist):
                .properties/ directory, INVARIANTS.md, CI check configs,
                type constraints, API contracts, schema definitions
2. [MECHANICAL] Run available mechanical checks:
                type checker, linter rules, contract tests
3. [MECHANICAL] For properties with clear pass/fail: evaluate and report result
4. [SURFACE]    For properties requiring judgment: "Generated code touches user 
                data. INVARIANT says 'all user-data endpoints require auth 
                middleware.' I added the auth wrapper — please verify this is 
                the correct middleware for this endpoint."
5. [MECHANICAL] Report: "Mechanically verified: N properties passed. 
                Flagged for human review: M properties (listed above)."

Why this is a superpower: The agent doesn't just generate plausible code. It checks its own output against declared properties before the human ever sees it. The human receives code that's already been evaluated against the team's safety boundaries — not just code that looks right.

What it looks like in practice:

Without skill:
    Agent generates API endpoint. Looks correct. Human merges.
    Endpoint returns PII without authentication. Discovered in production.

With skill:
    Agent generates API endpoint.
    Property check: "INVARIANT: All endpoints returning user data 
    require authentication middleware."
    Agent: "Generated endpoint does not include auth middleware. 
    Adding authentication wrapper before presenting."

Skill 3: Mechanical-Verify

Resolves Fallacy #3: AI can't verify AI

When it fires: When the agent needs to verify its own output.

What the agent does:

1. [MECHANICAL] Classify each property to verify:
                Type constraint → run compiler
                API contract    → run contract test
                Structural      → run linter/static analysis
                Universal       → run property-based test
                Subjective      → flag for human (NOT self-review)
2. [MECHANICAL] Run ALL mechanical checks. Collect results.
3. [SURFACE]    For subjective properties: "This error message says
                'invalid input.' Is that clear enough for your users, 
                or should it specify what's invalid?"
4. [MECHANICAL] Report: "Mechanically verified: [list with pass/fail].
                Human review needed: [subjective items listed above]."

Why this is a superpower: The agent stops pretending it can judge its own output. It runs every mechanical check available and presents the RESULTS, not its OPINION. For subjective properties, it doesn't self-review — it surfaces the question to the human. The agent knows what it can verify deterministically and what it can't.

What it looks like in practice:

Without skill:
    Agent reviews its own code: "This looks correct."
    The code has a subtle type mismatch the agent doesn't catch
    because it pattern-matches appearance, not logic.

With skill:
    Agent: "Running compiler... type mismatch on line 47:
    expected []byte, got string. Fixing before presenting."
    The compiler caught what the agent's self-review would miss.

Skill 4: Spec-Before-Code

Resolves Fallacy #4: Dropping review ≠ removing bottleneck

When it fires: Before writing any implementation, after the brainstorming/planning phase.

What the agent does:

1. [MECHANICAL] Read all specifications that govern the target module:
                module interface, API contract, database schema, ADRs, conventions
2. [MECHANICAL] Extract the list of constraints that apply
3. [SURFACE]    Present constraints to human: "Before I write code, these 
                constraints apply to this module: [list]. Are these correct? 
                Any I'm missing?"
4. [MECHANICAL] After human confirms: generate within those constraints
5. [MECHANICAL] After generating: verify output satisfies each constraint
                using available mechanical checks
6. [SURFACE]    If any constraint can't be mechanically verified: "I couldn't
                confirm compliance with [constraint]. Please review this 
                specific aspect."

Why this is a superpower: The agent doesn't just generate code and hope someone reviews it. It reads the existing specifications, confirms the constraints with the human, generates within those constraints, and verifies compliance. The human reviews the constraint list (small, fast) instead of the code (large, slow). The review moves to the right level — specifications for humans, code verification for the agent.

What it looks like in practice:

Without skill:
    Agent generates code. Human reviews 200 lines. Takes 45 minutes.
    Misses that the code uses exceptions instead of result types.

With skill:
    Agent: "Module conventions require result types for error handling
    (from ADR-007). Generating with result types."
    Human reviews: "Yes, those constraints are correct. Go ahead."
    Agent generates. Agent verifies against constraints. 
    Human reviews the 3-line constraint confirmation, not the 200-line implementation.

Skill 5: Output-Audit

Resolves Fallacy #5: Better context ≠ correct output

When it fires: After generation, specifically checking output against properties that AREN'T in the retrieved context.

What the agent does:

1. [MECHANICAL] After generating code, search project docs for architectural
                properties that apply but weren't in the original context:
                - ADRs mentioning timeout, authentication, PII, concurrency
                - CI check configurations
                - CLAUDE.md / CONVENTIONS.md constraints
2. [MECHANICAL] For each found property: check if the generated code 
                violates it using available mechanical tools
3. [SURFACE]    Present findings: "Found 3 architectural properties not in 
                my original context. Timeout policy (ADR-012) applies — 
                I added context.WithTimeout. PII handling policy applies — 
                please verify I'm not logging the user email on line 34.
                Encryption-at-rest policy does not apply to this code path."

Why this is a superpower: The agent compensates for its own context limitation. RAG retrieves documents that are semantically similar to the task. Architectural properties are semantically DISTANT from the code they govern. This skill explicitly searches for the properties that RAG would miss — because they live in ADRs, convention documents, and CI configurations that aren't similar to the implementation task in vector space.

What it looks like in practice:

Without skill:
    RAG retrieves correct API docs. Agent generates correct API call.
    Code makes synchronous external call without timeout inside a
    transaction. Timeout policy is in ADR-012, never retrieved.
    Connection pool exhausts in production.

With skill:
    Agent generates API call. Output-audit fires.
    Agent: "Checking timeout policies... Found ADR-012: 
    'All external calls require context.WithTimeout(5s).'
    Generated code lacks timeout. Adding before presenting."

Skill 6: Deletion-Aware

Resolves Fallacy #6: Generated code is a liability

When the agent fires: During implementation and refactoring tasks.

What the agent does:

1. [MECHANICAL] Before generating, search the codebase for existing 
                implementations of the same or similar functionality
2. [SURFACE]    If duplicates found: "Found 3 existing implementations 
                of date formatting: utils/dates.go, handlers/format.go, 
                api/helpers.go. Recommend consolidating to one. Which 
                should be the canonical version, or should I extract a 
                new shared function?"
3. [MECHANICAL] After human decides: implement the consolidation
4. [MECHANICAL] After completing any task, report additions AND deletions: 
                "Added 45 lines. Deleted 120 lines. Net: -75 lines."

Why this is a superpower: The agent actively shrinks the codebase. Instead of the default behavior (generate new code for every task), the agent searches for duplication, extracts shared functions, and deletes redundant implementations. The deletions-to-additions ratio improves. The maintenance burden decreases with each task instead of increasing.

What it looks like in practice:

Without skill:
    Five developers prompt agents for date formatting over a month.
    Five implementations exist. Bug found in one. Other four remain broken.

With skill:
    Agent: "Found existing formatDate() in utils/dates.go.
    Using existing implementation instead of generating new one.
    Also found two other formatDate variants in handlers/ and api/.
    Recommend consolidating to the utils/ version. Shall I refactor?"

Skill 7: Boundary-Read

Resolves Fallacy #7: Specs already exist, they're not new work

When it fires: At the start of every task, before any code generation.

What the agent does:

1. [MECHANICAL] Identify which module/package the task targets
2. [MECHANICAL] Read the module's existing boundaries:
                exported interface, import restrictions, API contract, 
                database schema, configuration schema
3. [MECHANICAL] Generate code that satisfies all identified boundaries
4. [MECHANICAL] After generating: run linter/depguard to verify 
                no boundary is violated
5. [SURFACE]    If the task REQUIRES changing a boundary: "This task 
                needs a new exported function in pkg/stave/. Changing 
                the public interface. Please confirm this is intended 
                — it affects all consumers of this package."
6. [MECHANICAL] Report: "Module has N boundaries. Implementation 
                satisfies all N. No cross-boundary imports."

Why this is a superpower: The agent treats existing specifications as first-class constraints instead of ignoring them. Most agents generate code that happens to match the module's style. This agent READS the interface definition, KNOWS what's exported and what isn't, and REFUSES to generate code that violates the boundary. The Parnas boundaries the team already built are finally enforced — by the agent itself.

What it looks like in practice:

Without skill:
    Agent generates code that imports an internal package from
    a different module. No linter catches it. The hexagonal
    architecture erodes silently.

With skill:
    Agent: "Module internal/core/ is restricted — depguard rule
    prevents imports from internal/app/. Generating without
    cross-boundary import. Using the public interface in pkg/stave/ instead."

Skill 8: Protocol-Sync

Resolves Fallacy #8: More agents ≠ more productivity

When it fires: At the start of every task, when multiple agents are working on the same codebase.

What the agent does:

1. [MECHANICAL] Read the shared specification repo:
                naming conventions, error handling strategy, retry policy,
                API contract versions, architecture decision records
2. [MECHANICAL] Check the specification VERSION — confirm it matches
                what other agents are reading (prevent split-brain)
3. [MECHANICAL] Generate code that conforms to ALL shared specifications
4. [MECHANICAL] After generating: verify conformance using linter rules
                and convention checks
5. [SURFACE]    If a decision isn't covered by any specification: 
                "This task requires choosing a serialization format for 
                the new event type. No convention covers this. Options: 
                JSON (consistent with existing events) or Protobuf 
                (consistent with the gRPC migration plan in ADR-015). 
                Which should I use? Should this become a new convention?"
6. [MECHANICAL] Report: "Conforming to spec version 2.4. All conventions 
                match. One uncovered decision flagged above."

Why this is a superpower: The agent doesn't make invisible architectural decisions. It reads the coordination protocols (shared specifications), follows them, and flags decisions that aren't covered. Multiple agents producing code that follows the same conventions, same error handling, same retry strategy — without any coordination meetings. The specifications are the protocols. The agents are the nodes. The distributed system is consistent.

What it looks like in practice:

Without skill:
    Agent A uses camelCase for JSON fields.
    Agent B uses snake_case.
    Integration breaks silently. Bug takes two days to trace.

With skill:
    Both agents read conventions.md at task start.
    Both generate camelCase JSON fields.
    Integration works on the first try.

What makes these different from "just prompting better"

These aren't prompt improvements. They're structural workflow changes with a clear boundary between agent action and human judgment:

Prompting:    "Remember to check for existing libraries"
              → The agent might or might not. Probabilistic.

Skill:        compose-first fires automatically before every
              implementation task. The agent MUST search before
              generating. Mandatory workflow, not suggestion.

BUT:          The agent searches [MECHANICAL].
              The human picks which library to use [SURFACE].
              The agent never decides "this library is fine" on its own.

This boundary separates these skills from Fallacy #3 (AI verifying AI). The agent does exhaustive, deterministic legwork — searching, reading, running checks, collecting results. The human makes judgment calls — which library, whether a constraint applies, whether an uncovered decision should become a new specification. The agent that crosses this boundary is automating judgment with correlated failure modes. The agent that respects it is doing the mechanical work that frees human judgment for where it matters.

The Superpowers framework proved this distinction. Skills aren't suggestions. They're mandatory workflows that fire based on triggers. The agent checks for relevant skills before any task. The skills execute as part of the agent's process, not as afterthoughts.

Each skill above has:

A trigger (when it fires)
A process (what the agent does, step by step)
A verification (how to confirm the skill executed correctly)
A report (what the agent tells the human)

This is the same structure Superpowers uses for brainstorming, TDD, and code review. The Fallacy resolutions fit the same framework — because they're the same kind of thing: structured workflows that prevent a known failure mode.

The compound effect

An agent running all eight skills simultaneously:

Searches for existing implementations before generating (Compose-First)
Reads module boundaries and specifications (Boundary-Read)
Reads shared conventions (Protocol-Sync)
Confirms constraints with the human (Spec-Before-Code)
Generates within all identified constraints
Runs mechanical verification (Mechanical-Verify)
Checks output against architectural properties not in context (Output-Audit)
Evaluates against declared properties (Property-Check)
Searches for redundant code to delete (Deletion-Aware)
Reports what was composed, generated, verified, and flagged

This agent produces less code, more capability, fewer violations, and better architectural coherence than an agent running without these skills — using the SAME model, the SAME context, the SAME prompts. The difference isn't the AI. It's the workflow around the AI.

The Fallacies identified what breaks. The skills fix it — not by making the human smarter, but by making the agent's process better. The engineer who installs these skills gives their agent the architectural judgment that the model doesn't have. The model provides the generation. The skills provide the discipline. The combination "AI-assisted development that works" looks like.

The Fallacies of GenAI Development: Index · #1 Faster Generation · #2-#8 linked from index.

Superpowers by Jesse Vincent — the agentic skills framework that proved agents can follow mandatory structured workflows. 209k stars. The skills above follow the same pattern.

Stave — the specification gate that implements Property-Check and Boundary-Read for cloud infrastructure. 2,662 safety invariants. Deterministic verification.

Fallacies of GenAI Development #8: More AI Agents Means More Productivity

Bala Paranj — Sat, 06 Jun 2026 12:31:34 +0000

This is the eighth and final post in a series on the false assumptions teams make when building with generative AI. The series began with the observation that the trough of disillusionment for AI-assisted development has arrived — not because AI is useless, but because eight false assumptions made the trough inevitable. This post covers the last assumption and closes the series.

The Fallacy

"If one AI agent gives us a 10x boost, ten agents will give us 100x."

Why it's tempting

The arithmetic feels irresistible. One agent generates code for the backend. Another generates the frontend. A third writes tests. A fourth handles database migrations. A fifth generates documentation. Each agent works in parallel. No meetings, waiting or coordination overhead. Pure throughput.

Leadership sees the potential: a five-person team with fifty agents has the output of a fifty-person team at the cost of a five-person team plus API credits. The scaling is linear. The economics are transformational.

And the early results confirm it. Each agent, working on its own, produces impressive output. The backend agent generates Go code. The frontend agent generates React components. The test agent generates test suites. Each agent, in isolation, looks like a 10x developer.

Why it's wrong

You've seen this problem before. It has a name. It's called distributed systems.

A distributed system is a collection of independent actors that must coordinate to produce a coherent result. Each actor makes decisions locally. The system's correctness depends on those local decisions being compatible globally. When they aren't, you get inconsistency, conflicts, data corruption, and cascading failures.

AI agents working on the same codebase are a distributed system. Each agent makes decisions — variable names, error handling strategies, retry policies, data formats, abstraction levels, dependency choices. Each decision is made locally, in the context of one prompt, one file, one task. No agent sees the full picture. No agent coordinates with the other agents. Each agent's decisions are invisible to the others.

Distributed systems engineers spent 40 years learning that you can't scale a distributed system by adding more nodes. You can only scale it by adding protocols — consensus mechanisms, ordering guarantees, conflict resolution rules, interface contracts. Without protocols, more nodes means more conflicts, not more throughput.

The same applies to AI agents. More agents without specifications means more invisible decisions, more inconsistency, more cognitive fragmentation — not more productivity.

The boom

Month 1-2: The parallel sprint. Five agents work simultaneously on different parts of the system. Each produces well-structured code. PRs flow in from every direction. The team merges them rapidly. The system takes shape faster than anyone expected.

Month 3-4: The integration cracks. The backend agent chose camelCase for JSON field names. The frontend agent expected snake_case. The database migration agent used PascalCase for column names. None knew about the others' choices. Each was reasonable in isolation. The integration fails silently — data flows through but field mappings are wrong. The bug appears as "the UI shows the wrong value" and takes two days to trace to a naming mismatch across three layers.

Month 5-6: The contradictory patterns. The backend agent implemented retries with exponential backoff and jitter. The API gateway agent implemented retries with a fixed 3-second delay. Both are valid retry strategies. Both were generated from different training data patterns. When a downstream service is slow, the backend retries with increasing delays while the gateway retries every 3 seconds — creating a thundering herd that overwhelms the already-slow service. The agents' patterns contradicted each other. Neither knew the other existed.

Month 7-8: The architectural drift. Each agent, given different tasks over months, evolved different internal patterns. The backend agent started using result types for error handling. The frontend agent uses exceptions. The test agent mixes both depending on which prompt it received. The codebase has three error handling philosophies, each locally consistent, globally incoherent. A new developer opens the codebase and can't determine which pattern is correct because all three exist in production.

Month 9-10: The edit war. Agent A refactors a shared utility function for performance. Agent B, in a separate task, refactors the same function for readability. Agent A's change merges first. Agent B's change overwrites A's optimization. Neither agent knows the other touched the file. The team pays for the token cost of both refactors and gets the result of neither. Adam Bender of Google has a name for this — in his talk Software Engineering at the Tipping Point, he calls it the agentic edit war: two agents refactoring the same file toward different goals, livelocking the system while the company pays for the tokens on both sides.

Worse, without a pattern specification, the cycle repeats. Agent A sees the unoptimized code and refactors it again for performance. Agent B sees the unreadable code and refactors it again for readability. The agents consume tokens infinitely, bouncing the code back and forth between two valid-but-contradictory goals. In distributed systems, this is called livelock — the system is active but making no progress. In AI development, it's called your API bill.

Month 11-12: The coordination collapse. The team needs to make an architectural change — migrate from REST to gRPC. Each agent needs to be told individually. Each interprets the migration prompt differently. The backend agent generates gRPC server code. The frontend agent keeps generating REST client calls because its prompt wasn't updated. The test agent generates tests for both protocols because it sees both in the codebase. The migration that should take a week takes two months because every agent is working against the others.

The team discovers they've been managing a distributed system without the protocols that distributed systems require. Each agent is a node. Each node is making decisions. Nobody built the consensus layer.

What distributed systems already solved

Hold up the two systems side by side:

Distributed Computing (1990s):           Distributed AI Development (2020s):
───────────────────────────              ──────────────────────────────────
Multiple processes on multiple nodes     Multiple agents on one codebase
Each makes local decisions               Each makes local decisions
No shared state by default               No shared context by default
Inconsistency is the default outcome     Inconsistency is the default outcome

Distributed computing solved this with protocols:

Protocol                    What it solves              AI equivalent
────────────────────        ────────────────────        ──────────────────────
Consensus (Paxos, Raft)     Agreement on shared state   Shared specification repo
Ordering (vector clocks)    Event sequencing            Architectural priority rules
Conflict resolution         Concurrent modifications    Specification gate on merge
Interface contracts (IDL)   Cross-service compatibility API contracts + contract tests
Schema evolution            Backward compatibility      Migration specifications
Circuit breakers            Cascading failure prevention Dependency scope limits

Every protocol in the left column has an AI development equivalent in the right column. The solutions exist. They're called specifications, contracts, and enforcement gates. They're the same coordination mechanisms — applied to agents instead of processes.

The five coordination failures and their fixes

1. Naming inconsistency → Convention specification

Problem: Each agent chooses its own naming conventions.
Fix:     A convention specification fed to every agent as context.
         "JSON fields: camelCase. Database columns: snake_case.
          Go structs: PascalCase. Environment variables: UPPER_SNAKE."
         Four lines. Every agent reads them. Every change is checked
         against them by a linter. Inconsistency becomes mechanically
         impossible.

         Critical: the specification must be immutable for the duration
         of concurrent agent tasks. In distributed systems, "split brain"
         happens when nodes have different versions of the truth. If
         Agent A has the old naming convention and Agent B has the new
         one, the codebase gets both. Version the specification. Update
         it between task batches, not during them.

2. Pattern contradictions → Architectural decision records

Problem: Each agent implements different patterns for the same concern.
Fix:     One ADR per cross-cutting concern.
         "ADR-007: Retry strategy is exponential backoff with jitter,
          base 100ms, max 5 attempts, across ALL services."
         The specification is the protocol. The CI check is the
         enforcement. Any agent that generates fixed-delay retries
         fails the build.

3. Edit conflicts → Ownership boundaries

Problem: Two agents modify the same file with contradictory goals.
Fix:     CODEOWNERS + module boundaries.
         Each module has one owner (human or agent). Cross-module
         changes require the interface contract to be satisfied.
         Agents can't modify modules outside their scope.
         Same principle as microservice boundaries — but for agents.

4. Error handling divergence → Interface contracts

Problem: Three error handling philosophies coexist in the codebase.
Fix:     One interface contract: "All public functions return
         (result, error). No exceptions. No panics. No sentinel values."
         Enforced by the compiler (Go) or by a linter rule (TypeScript).
         The contract is the specification. The tool is the enforcement.

5. Migration incoherence → Change specifications

Problem: An architectural migration is interpreted differently by each agent.
Fix:     A migration specification: "Phase 1: Add gRPC endpoints alongside
         REST. Phase 2: Migrate clients to gRPC. Phase 3: Remove REST.
         Current phase: 1. Agents must not remove REST endpoints."
         Each agent reads the current phase. The specification prevents
         agents from jumping ahead or falling behind.

Each fix is small. Each is a few lines of text. Each is fed to agents as context AND enforced mechanically by CI. The context helps agents make correct decisions. The enforcement catches them when they don't.

The principle: specifications are protocols for agents

In distributed computing, protocols enable coordination without requiring every node to understand every other node. Node A doesn't need to know Node B's implementation. It needs to know Node B's interface contract. If both nodes respect the protocol, the system is consistent — regardless of how many nodes you add.

The same principle applies to AI agents. Agent A doesn't need to know Agent B's prompt. It needs to know the specification that governs the module it's working on. If all agents respect the specifications, the codebase is consistent — regardless of how many agents you add.

Distributed computing:     Protocol enables coordination between nodes
Distributed AI dev:        Specification enables coordination between agents

Distributed computing:     More nodes + same protocol = more throughput
Distributed AI dev:        More agents + same specifications = more productivity

Distributed computing:     More nodes + no protocol = more conflicts
Distributed AI dev:        More agents + no specifications = more inconsistency

Scaling agents is scaling a distributed system. The mechanisms are the same. The lesson is the same. The solution is the same.

The series, complete

Eight fallacies. One meta-pattern.

Fallacy #1: Faster generation = faster engineering
            → The leading sub-system outran the lagging ones

Fallacy #2: Looks correct = is correct
            → Plausibility is not correctness

Fallacy #3: AI can verify AI
            → Correlated failure modes don't converge

Fallacy #4: Drop review = remove bottleneck
            → Removing a gate without replacing it removes the safety net

Fallacy #5: Better context = correct output
            → Input quality doesn't guarantee output correctness

Fallacy #6: Generated code is an asset
            → Code is a liability; capability is the asset

Fallacy #7: Specs are new work
            → The specifications already exist; the enforcement doesn't

Fallacy #8: More agents = more productivity
            → More actors without protocols = more conflicts

Every fallacy stems from one root assumption: generating the output is the hard part.

This assumption is wrong. Understanding the output, verifying it, maintaining it, coordinating the actors that produce it, and preserving the rationale for why it's shaped this way — those are the hard parts. They always were. AI made the easy part faster. The hard parts didn't change.

Peter Deutsch's Distributed Computing Fallacies worked because they named the assumptions that every developer made, discovered were wrong, and paid for in production. The network is not reliable. Latency is not zero. Bandwidth is not infinite.

The Fallacies of GenAI Development work the same way. Generation is not engineering. Plausible is not correct. More agents is not more productivity. Each assumption sounds true. Each leads to failure. Each has already been resolved by a domain that learned the lesson first.

The resolution to all eight

The resolution is the same across every fallacy, every domain, every era:

Recognize the specifications that already exist in your system — types, contracts, schemas, boundaries (Fallacy #7). Enforce them mechanically on every change, at machine speed (Fallacies #1, #3, #4). Verify the output against declared properties, not just the input quality (Fallacies #2, #5). Measure properties verified, not code generated (Fallacy #6). Use specifications as coordination protocols for agents (Fallacy #8).

One architectural principle. Eight applications. The teams that adopt it first will emerge from the trough of disillusionment ahead of everyone else. The teams that don't will learn the same lessons the hard way — the same way distributed systems developers learned Deutsch's fallacies, one production incident at a time.

The engineer's role has changed. Not from "writing code" to "writing prompts" — that's the Fallacy #4 trap, the prompt operator with no ownership. The real shift is from writing code to designing protocols. The specifications, contracts, boundaries, and enforcement gates that enable agents to coordinate safely. The engineer becomes the protocol designer for a distributed system of AI actors. That's not a demotion from "programmer." It's the same move the industry made when it went from writing assembly to designing systems. The abstraction level changed. The engineering judgment became more valuable, not less.

The trough is real. The exit is specifications — recognized, enforced, and verified.

This concludes The Fallacies of GenAI Development. The complete series: #1 Faster Generation ≠ Faster Engineering · #2 Plausible ≠ Correct · #3 AI Can't Verify AI · #4 Dropping Review ≠ Removing Bottleneck · #5 Better Context ≠ Correct Output · #6 Generated Code Is a Liability · #7 Specifications Already Exist · #8 More Agents ≠ More Productivity

For cloud infrastructure specifically, the specification-first model is implemented in Stave — an open-source tool that evaluates configuration snapshots against 2,662 safety invariants with deterministic mechanical verification. Apache 2.0.

For a single IAM policy file, try iam-explain — point it at your policy JSON, see what the math says. The specifications are already in your policy. The tool shows you what they mean.

The Fallacies of GenAI Development were inspired by Peter Deutsch's "Fallacies of Distributed Computing" (1994). The resolution draws from Parnas (1972), Altshuller's TRIZ (1946), Byron Cook's automated reasoning at AWS, and evidence from aviation, nuclear operations, financial trading, and Google's monorepo. Each fallacy was discovered independently across domains. The convergence is the evidence.

Fallacies of GenAI Development #7: Specifications Are a New Artifact You Have to Create

Bala Paranj — Fri, 05 Jun 2026 11:25:42 +0000

This is the seventh in a series of eight posts on the false assumptions teams make when building with generative AI. Fallacy #1 through #5 covered why AI-speed development breaks without verification. Fallacy #6 covered why generated code is a liability, not an asset. This post covers the assumption that stops most teams from adding verification: the belief that it requires writing new specifications from scratch.

The Fallacy

"To adopt specification-first development, we need to write specifications for everything. That's too much work."

Why it's tempting

The word specification carries baggage. Engineers who've been in the industry long enough remember:

The 1980s: Formal methods. Z notation. VDM. B method. Full behavioral specifications that were as complex as the implementation. Writing the spec took as long as writing the code. Maintaining both was double the work. When deadline pressure hit, the spec was the first thing dropped.

The 1990s: IEEE 830-style requirements documents. Hundreds of pages. Maintained alongside the code by a dedicated requirements engineer. The document and the code drifted apart within months. The requirements document became fiction that nobody read.

The 2000s: Model-driven development. UML diagrams generated code. The diagrams were the specification. Maintaining the diagrams was a full-time job. When the generated code needed manual modifications, the diagrams and the code diverged permanently.

Each attempt failed for the same reason: the specification tried to describe EVERYTHING the system does. A complete behavioral specification is as complex as the implementation. Maintaining both is unsustainable. The spec rots. The team abandons it. The effort was wasted.

Engineers hear specifications and think of these failures. The reaction is immediate: "We tried that. It didn't work. Too much overhead. We're not doing it again."

Why it's wrong

The fallacy is in the assumption that specification means what it meant in the 1980s — a complete behavioral description of the system.

It doesn't. Look at your codebase right now. You already have specifications. You just don't call them that.

The specifications you already maintain

Type signatures. Every function in a typed language has a specification: what it accepts and what it returns. func ParsePolicy(path string) (*PolicyDocument, error) — that's a specification. It says: this function takes a string, returns a document or an error. Any code that calls it with the wrong type fails at compile time. You already write these. You already maintain them. You already enforce them mechanically — the compiler is the enforcement gate.

You're already using formal methods. Type checking is the most successful formal verification tool in history. It proves a mathematical property about your code — type safety — every time you compile. It's fast, deterministic, and catches errors before any human sees them. We have to apply the success of the compiler to your other boundaries.

API contracts. Your REST API has an OpenAPI spec. Your gRPC services have Protocol Buffer definitions. Your GraphQL API has a schema. Each one specifies: these are the endpoints, these are the fields, these are the types, these are the required parameters. Any client that violates the contract gets an error. You already write and version them.

Database schemas. Your migration files specify: these are the tables, these are the columns, these are the constraints, these are the foreign keys. NOT NULL, UNIQUE, FOREIGN KEY — each is a specification enforced by the database engine on every write. You already write these.

Configuration schemas. Your Kubernetes manifests specify resource limits, health checks, replica counts. Your Terraform files specify infrastructure state. Your CI pipeline files specify build steps and their dependencies. Each is a specification that a machine reads and enforces.

Interface boundaries. Your Go packages have exported and unexported functions. Your Java classes have public and private methods. Your Python modules have __all__ lists. Each boundary specifies: this is the interface, everything else is hidden. Parnas told you to create these in 1972. You did.

What Parnas said

David Parnas's 1972 paper "On the Criteria To Be Used in Decomposing Systems into Modules" argued that humans can't hold entire systems in their heads. The solution: information hiding. Each module exposes an interface (what it does) and hides its implementation (how it does it).

This paper is the foundation of every modular programming language, every API boundary, every microservice architecture, every package system. When you write a Go package with exported functions, you're implementing Parnas. When you define a Protocol Buffer service, you're implementing Parnas. When you create a database schema with foreign key constraints, you're implementing Parnas.

You've been writing specifications your entire career. You just called them interfaces and schemas and contracts and types.

What's missing

The specifications exist. What's missing is a single thing: mechanical enforcement across all of them, on every change, at AI speed.

Your type system enforces type specifications — but only within one language, one compilation unit. It doesn't enforce that the TypeScript client matches the Go server's API contract.

Your database engine enforces schema constraints — but only at write time. It doesn't enforce that the application code handles all the constraint violations correctly.

Your API contract exists — but most teams don't have a CI check that fails the build when generated code deviates from the OpenAPI spec.

Your module boundaries exist — but nothing prevents an AI agent from generating code that reaches across boundaries through reflection, type casting, or dynamic imports.

The specifications are there. The enforcement is human — code review, manual testing, "I'll check that during review." And as Fallacy #4 showed, human enforcement doesn't scale to AI-speed generation.

There's an irony here that connects to the GenAI context: AI models are remarkably good at following well-defined contracts and remarkably bad at guessing implicit ones. When your OpenAPI spec is enforced mechanically, you give the AI a fixed target. It doesn't have to guess your intent. It just has to satisfy the contract. The mechanical enforcement that catches human errors ALSO catches AI errors — and the AI benefits from the explicit boundary even more than the human does, because the AI has zero institutional knowledge to fall back on when the contract is absent.

The three-layer model

Every domain that faced the "system too complex for humans to understand" problem arrived at the same three-layer structure:

Layer 1 — Boundaries (Parnas, 1972):
    The specifications. Already in your codebase.
    Module interfaces, API contracts, type signatures, schemas.
    Defines WHAT each part promises to the rest.

Layer 2 — Mechanical enforcement (the missing layer):
    A gate that checks EVERY change against EVERY boundary.
    At machine speed. Before deployment.
    The type checker is Layer 2 for type specifications.
    CI contract tests are Layer 2 for API specifications.
    What's missing: Layer 2 for architectural and security specifications.

Layer 3 — Rationale preservation:
    WHY each boundary exists.
    The failure mode it prevents. The incident that motivated it.
    Connected to the boundary so that changing it triggers
    review of the rationale.

Layer 1 you already have. Your interfaces, contracts, types, and schemas.

Layer 2 you partially have. The compiler is Layer 2 for types. The database engine is Layer 2 for schemas. What you're missing: Layer 2 for the architectural constraints, security properties, and cross-service contracts that currently live in people's heads or in documents.

Layer 3 you barely have. Maybe some Architecture Decision Records. Maybe some comments explaining "why" in the code. Rarely connected to the boundaries they explain. Rarely enforced — nobody checks whether a proposed change to an interface violates the rationale that shaped it.

The extreme domains added Layer 2

No safety-critical domain stopped at Layer 1. Every one discovered that boundaries without enforcement don't survive contact with reality at scale.

Microprocessor design: Interface specifications exist (bus protocols, timing diagrams). Formal verification tools CHECK every circuit against the specifications mechanically. You can't tape out a chip without passing the verification suite.

Nuclear operations: Operating procedures exist (the specifications). Physical interlocks ENFORCE parameter limits mechanically. The reactor scrams automatically when parameters exceed the envelope — regardless of what the operator does.

Aviation: Flight envelopes exist (the specifications). Fly-by-wire systems ENFORCE them mechanically. The pilot can't stall the aircraft — the computer overrides the input before it reaches the control surfaces.

Google monorepo: API contracts exist (the specifications). CI gates ENFORCE them on every commit. A large-scale change touching millions of lines of code merges only when every affected contract test passes mechanically.

Each domain: specifications existed (Layer 1). Human enforcement couldn't keep up (Layer 2 missing). They added mechanical enforcement (Layer 2). The system survived scaling.

Your codebase is in the same position. Layer 1 exists. Layer 2 is partial. AI-speed generation made the gap visible.

Why this isn't the 1980s

The formal methods failures of the 1980s had a specific cause: the specification was a COMPLETE BEHAVIORAL DESCRIPTION — as complex as the code. That's not what this article is proposing.

1980s approach (failed):
    Write a full specification of everything the system does.
    Spec complexity ≈ code complexity.
    Maintaining both is double the work.
    Spec rots. Team abandons it.

Current approach (what already exists + enforcement):
    Recognize the specifications you already maintain.
    Add mechanical enforcement for the ones that aren't enforced.
    Spec complexity << code complexity because these are
    INTERFACE specifications, not BEHAVIORAL specifications.
    Types, contracts, schemas — small, stable, already maintained.

The type signature func(string) (*Document, error) is not a complete behavioral specification of the function. It's an interface specification — what goes in, what comes out. It's smaller than the code. It changes less often. The compiler already enforces it.

The OpenAPI spec is not a complete description of your API's behavior. It's an interface specification — endpoints, fields, types, required parameters. It's smaller than the code. It changes less often. Contract testing already enforces it.

The database schema is not a complete model of your data's semantics. It's a structural specification — tables, columns, constraints. It's smaller than the application code. It changes less often. The database engine already enforces it.

Each of these succeeded where the 1980s failed because they specify INTERFACES, not BEHAVIOR. Interfaces are small. Behavior is large. Small specifications are maintainable. Large specifications rot.

What you can do this week

1. Inventory your existing specifications. Open your codebase. Count: How many typed function signatures? How many API contracts (OpenAPI, Protobuf, GraphQL schemas)? How many database schemas with constraints? How many CI pipeline definitions? You'll find more specifications than you expected. You've been writing them your entire career.

2. Identify the unenforced specifications. Of the specifications you found, which ones have mechanical enforcement (compiler, contract test, schema validator, CI check)? Which ones exist as documents but aren't checked on every change? The unenforced ones are your Layer 2 gaps. Each gap is a place where AI-generated code can violate an interface without anyone catching it.

3. Enforce one specification mechanically this week. Pick the most critical unenforced specification. Your OpenAPI contract that exists but has no contract test? Add one. Your module boundary that exists but nothing prevents cross-boundary imports? Add a linter rule — tools like arch-go, depguard, or eslint-plugin-import can enforce that your internal packages can't be imported by your public packages. That's a Layer 2 gate for a Layer 1 boundary, implemented in one config file. Your architecture decision that exists as a document but has no CI check? Write one check for one rule.

One specification. Already written. One enforcement gate. New this week. That's the entire adoption path. Not "write specifications for everything." Not "adopt formal methods." Not "add a six-month process change." One existing specification. One new CI check. This week.

4. Attach one rationale. Take the specification you just enforced. Add one comment or one metadata field: WHY does this boundary exist? What incident or failure mode motivated it? When someone proposes changing it next quarter, the rationale will save the team from reintroducing the problem the boundary was designed to prevent.

A Layer 3 comment isn't // This parses JSON. It's:

// We use this specific parser because it handles the non-standard
// date format from the legacy billing system. The standard library
// parser silently truncates these dates, causing invoice mismatches.
// See Incident #402. Do not replace without verifying billing
// integration tests pass with the legacy date format.

Without this, an AI agent will optimize the code by replacing the parser with the standard library version. The tests pass — because the test fixtures use standard dates. The billing integration breaks in production — because real billing data uses the legacy format. The rationale prevents the optimization that looks correct but isn't.

The specifications already exist. Parnas gave you the principle. Your language gave you the type system. Your API framework gave you the contract. Your database gave you the schema. The only new thing is the enforcement gate — and that's a CI configuration, not a cultural transformation.

Next and final in the series: **Fallacy #8 — "More AI Agents Means More Productivity."* Why adding agents without adding specifications is like scaling a distributed system without adding protocols — and why the same coordination mechanisms that solved distributed computing solve distributed AI development.*

The Fallacies of GenAI Development: eight assumptions every team is making. Each one leads to an architectural failure. Each one has already been solved.

Fallacies of GenAI Development #6: AI-Generated Code Is an Asset

Bala Paranj — Wed, 03 Jun 2026 10:28:19 +0000

This is the sixth in a series of eight posts on the false assumptions teams make when building with generative AI. Fallacy #1 covered the generation-engineering gap. Fallacy #2 covered plausible vs. correct. Fallacy #3 covered AI verifying AI. Fallacy #4 covered removing review. Fallacy #5 covered context vs. verification. This post covers the assumption that generated code is value — when it's actually cost.

The Fallacy

"The AI generated 10,000 lines this week. We're 10x more productive."

Why it's tempting

Productivity has always been hard to measure in software. Lines of code was a bad metric, but it was a metric. When AI made code generation fast, the metric became irresistible again. The team generated more code. The PRs are larger. The features ship faster. The graphs go up and to the right.

Leadership loves it. More output. More features. More velocity. The investment in AI tooling is paying off — look at the numbers. The team is producing more than ever.

And it feels good to the engineers too. You describe a feature. The AI generates the implementation. You ship it. The dopamine hit of productivity is real. You built many things today. The backlog is shrinking. The sprint velocity is through the roof.

Why it's wrong

Jeff Atwood said it plainly: code is a liability, not an asset. The concept originates in Lean Manufacturing, applied to software by Mary and Tom Poppendieck (2003): unshipped code is Work in Progress waste, and shipped code is a maintenance burden. AI didn't change the economics — it made it faster to create the liability.

Every line of code is something you have to:

Compile. More code means longer build times. Bigger binaries.
Test. More code means more test cases needed. Dependencies grow quadratically — 10x code may mean 100x test compute.
Debug. More code means more potential failure points. More places for bugs to hide.
Secure. More code means more attack surface. More code paths to audit.
Understand. More code means more cognitive load. More context to hold when making changes.
Maintain. More code means more things that break when dependencies change. More migration work when frameworks evolve.

Studies by the Consortium for Information & Software Quality (CISQ) put numbers to this: the cost of developing code is typically only 20–30% of its total lifecycle cost. The remaining 70–80% is maintenance. If AI makes development 10x faster but doesn't change the maintenance profile, the total lifecycle savings are negligible — only a 10% reduction of the 30% development slice. If the AI increases code volume, the total cost of ownership actually rises, even if development was free.

The team that generated 10,000 lines this week didn't create 10,000 lines of value. They created 10,000 lines of ongoing cost. The value was in the feature. The cost is in the code. The feature could have been delivered with 1,000 lines if the right abstractions existed. The 9,000 extra lines are pure liability.

The boom

Month 1-3: The output surge. The team generates more code than ever. Features ship. The backlog shrinks. Sprint velocity doubles, then triples. Leadership presents the metrics at the quarterly review. AI is working.

Month 4-6: The build slows down. Compilation time increases. CI pipeline takes longer. Developers wait for builds. The wait time is small at first — 10%, 15% longer. Nobody notices because the generation is so fast. But the trend is upward.

Month 7-9: The test burden compounds. New code depends on existing code. Existing code depends on other existing code. The dependency graph grows quadratically. A change to one module triggers tests in 50 other modules. The test suite that ran in 8 minutes now takes 45 minutes. Developers start skipping test runs locally and relying on CI. CI queues back up.

Winters, Manshreck, and Wright document this in Software Engineering at Google (2020). Adam Bender names it precisely: "If your codebase is 10 times larger and you're trying to test all the dependencies so that you're sure nothing will break, you may have upwards of 100 times as many tests running. Maybe 1,000 times as many tests. That's going to be a line item in your budget."

Month 10-12: The maintenance cliff. A dependency needs updating. A security patch requires changes. A framework is deprecated. Each change touches thousands of generated lines that nobody fully understands. The team that generated code in minutes spends weeks migrating it. The code that was cheap to produce is expensive to maintain. Lehman's Second Law of Software Evolution (1980) predicts this: as a program is continually changed, its complexity increases unless active work is done to reduce it. AI is purely additive — it adds structure-deteriorating code at high speed. Without deliberate refactoring and deletion, the system reaches the maintenance cliff faster than human teams can recover.

There's a subtler rot underneath. Five developers each generated their own version of formatDate(), parseUserInput(), and retryWithBackoff(). Each used a different prompt. Each produced a slightly different implementation. A bug is found in one version of parseUserInput() — but nobody knows four other versions exist scattered across the codebase. The fix patches one. The other four remain broken. This is inconsistency debt — the invisible cost of generation without reuse. A library would have had one implementation. Five generations created five liabilities. Google's "One Version Rule" (documented in Software Engineering at Google) exists precisely to prevent this: every library has a single version across the monorepo. AI generation is the ultimate violator of the One Version Rule — each prompt produces a "new version" of a common utility, creating dependency hell within a single codebase.

And then the quiet realization: the team isn't building new features anymore. They're maintaining old code. The AI generates new code faster than ever. But the team spends most of their time on the code that already exists — patching, migrating, debugging, securing. The generation speed is irrelevant when the maintenance burden consumes all available time.

The ratio inverts. In month 1, the team spent 90% of their time building new features and 10% maintaining existing code. By month 12, it's 30% building and 70% maintaining. The AI made the 30% faster. Nobody made the 70% faster. The overall velocity DECREASES even as the generation speed INCREASES.

The unit of progress is wrong

The software industry has been optimizing for the wrong unit of progress for decades. AI made the mistake faster.

Code generation:     More lines → more volume → more liability
Function composition: Same functions, composed differently → more capability → zero new liability

Unix proved the right unit 50 years ago. Doug McIlroy, the inventor of Unix pipes, summarized the philosophy in 1978: "Expect the output of every program to become the input to another, as yet unknown, program." grep, sort, uniq, awk, sed — each is a function with a stable interface (stdin/stdout). They've survived:

The transition from mainframes to minicomputers to PCs to servers to cloud
Six generations of operating systems
Dozens of programming language fashions
Every deployment paradigm from tape to containers

They survived because the unit of reuse is the function, not the code. Nobody regenerates grep for each project. Nobody writes a new sorting algorithm for each application. You compose existing functions through stable interfaces. The value is in the composition. The functions are reusable. The liability is near zero because you're not maintaining them, it is the job of the upstream maintainers. AI generation is an anti-Unix force: it generates custom, non-standard implementations every time instead of reusing standard tools.

Google proved the same principle at organizational scale. When a team needs an RSS feed generator, they reuse the existing RSS function. They don't generate new code that reimplements RSS parsing. Google's internal codebase has millions of reusable functions with stable interfaces. The productivity comes from knowing which function to use, not from generating a new implementation.

The metric inversion

The fallacy persists because teams measure the wrong thing:

What teams measure:          What they should measure:
─────────────────────        ───────────────────────────
Lines of code generated      Properties verified per module
PRs merged per sprint        Specification coverage (% of behaviors governed)
Features shipped             Functions reused vs. code generated
Sprint velocity points       Maintenance burden (% time on existing vs. new)

The left column goes up when you generate more code. Leadership celebrates. The right column would show that more generated code means LOWER specification coverage (more ungoverned behavior), LOWER reuse ratios (more reimplemented functions), and HIGHER maintenance burden (more time on existing code).

The team that generated 10,000 lines and verified zero properties is more productive on the left-column metrics and more indebted on the right-column metrics. The left column measures output speed. The right column measures engineering health. When these diverge — output going up, health going down — the team is building debt, not value.

The resolution: compose, don't generate

The alternative to generating more code is composing fewer, better-verified units.

GENERATE (current model):
    Need an RSS parser    → AI generates 200 lines
    Need a date formatter → AI generates 80 lines
    Need an HTTP client   → AI generates 150 lines
    Need JSON validation  → AI generates 120 lines

    Total: 550 lines of new code to maintain
    Properties verified: 0
    Reuse potential: 0 (tightly coupled to this project)

COMPOSE (alternative model):
    Need an RSS parser    → import rss-parser (maintained upstream)
    Need a date formatter → import date-fns (maintained upstream)
    Need an HTTP client   → import http-client (maintained upstream)
    Need JSON validation  → validate against JSON Schema (maintained upstream)

    Total: 4 import statements + 20 lines of composition
    Properties verified: each library has its own test suite
    Reuse potential: 100% (same libraries across every project)

The composed version has 96% less code. 96% less to compile, test, debug, secure, understand, and maintain. The libraries are maintained by their upstream communities. The composition is the only new code — and it's 20 lines, reviewable in minutes.

AI's best role isn't generating the 550 lines. It's helping you FIND the right four libraries and COMPOSE them correctly. The generation is the least valuable part. The selection and composition are the most valuable parts.

The expert AI-assisted engineer isn't a faster typist. They're a better librarian. Their value isn't in generating a custom implementation of a common pattern. It's in knowing that an optimized, well-tested, actively-maintained library already exists — and using the AI to write the 20 lines of glue code to connect it. The librarian ships less code and more capability. The typist ships more code and more debt. Caldiera and Basili (1991) demonstrated this in their research on software reuse: successful software evolution depends on prioritizing reuse over creation. AI makes creation so cheap that it crowds out the impulse to search for existing libraries — a phenomenon engineers recognize as "Not Invented Here Syndrome," now running at machine speed.

When generation is appropriate

Generation is appropriate when:

No reusable function exists. Genuinely novel logic with no upstream library. Generate it — but immediately extract it as a reusable function with a stable interface, tests, and a specification. Don't leave it inline.

The code is disposable. Prototypes, experiments, one-off scripts. Generate freely. But don't let disposable code become production code. Bender's isolation principle: "You don't want that cool prototype code to find its way into production."

The composition IS the code. Glue code that connects well-tested components. This is appropriate for generation because the value is in the wiring, the individual components are already verified, and the glue code can be verified against the interface contracts of the components it connects.

In each case, the generated code should be the MINIMUM necessary — not the maximum the AI can produce. The AI that generates the least code to solve the problem is more valuable than the AI that generates the most. Less code = less liability = less maintenance = more time for new capabilities.

The Delete Code Test

If your team generated 10,000 lines last week and you deleted 9,000 of them, replacing them with imports and 200 lines of composition — would the system work the same way?

If yes, you generated 9,000 lines of pure liability.

If you're not sure, you don't have enough specifications to know — which is Fallacy #2 (plausible ≠ correct) and Fallacy #5 (context ≠ verification) revisiting you.

The goal isn't more code. It's more CAPABILITY with LESS CODE. Every line you didn't generate is a line you don't maintain. Every function you reused instead of regenerated is a function someone else maintains. Every specification you verified is a property that holds regardless of how much code is behind it.

Code is a liability. Less of it is better. The most productive team isn't the one that generates the most. It's the one that ships the most capability while maintaining the least code.

What you can do this week

1. Measure your generation-to-reuse ratio. For your last 10 AI-generated files, count: how many imported existing libraries vs. how many reimplemented functionality that libraries already provide? If reimplementation exceeds reuse, the AI is generating liability.

2. Before generating, ask: "Does this function already exist?" Better yet, ask the AI: "Is there a popular, well-maintained open-source library that handles [Task X]?" BEFORE asking it: "Write a function that handles [Task X]." If a library exists, the AI's job is to write the integration code, not the logic. The 5-minute search saves weeks of future maintenance.

3. Track maintenance burden monthly. What percentage of your team's time goes to maintaining existing code vs. building new capabilities? If that percentage is rising while code generation is also rising, the generation is feeding the maintenance burden. The metric tells you whether your AI investment is producing value or producing debt.

4. Measure your deletions-to-additions ratio. A high-performing AI-assisted team should be deleting redundant code and replacing it with better abstractions as often as they add new code. If your ratio is 10:1 additions-to-deletions, the codebase is growing without consolidation. If it's closer to 2:1 or even 1:1, the team is composing — replacing generated sprawl with verified, reusable units. The ratio tells you whether the AI is building capability or building debt.

The AI is a remarkable code generator. That's the problem. The skill isn't generating more code. It's knowing when NOT to generate — when to compose, when to reuse, when to import, and when to write the 20 lines of composition instead of the 550 lines of reimplementation.

Next in the series: **Fallacy #7 — "Specifications Are a New Artifact You Have to Create."* Why the specifications already exist in your codebase, why Parnas told you to create them in 1972, and why the only new thing is mechanical enforcement.*

The Fallacies of GenAI Development: eight assumptions every team is making. Each one leads to an architectural failure. Each one has already been solved.

References

Caldiera, G. and Basili, V. (1991). "Identifying and Qualifying Reusable Software Components." IEEE Computer, 24(2).
CISQ (Consortium for Information & Software Quality). "The Cost of Poor Software Quality in the US."
Lehman, M.M. (1980). "Programs, Life Cycles, and Laws of Software Evolution." Proceedings of the IEEE, 68(9).
McIlroy, M.D. (1978). "Unix Time-Sharing System: Foreword." The Bell System Technical Journal, 57(6).
Poppendieck, M. and Poppendieck, T. (2003). Lean Software Development: An Agile Toolkit. Addison-Wesley.
Winters, T., Manshreck, T., and Wright, H. (2020). Software Engineering at Google: Lessons Learned from Programming Over Time. O'Reilly Media.

Fallacies of GenAI Development #5: Better Context Prevents Hallucination

Bala Paranj — Wed, 03 Jun 2026 10:28:19 +0000

This is the fifth in a series of eight posts on the false assumptions teams make when building with generative AI. Fallacy #1 covered the generation-engineering gap. Fallacy #2 covered plausible vs. correct. Fallacy #3 covered AI verifying AI. Fallacy #4 covered removing the review gate. This post covers the assumption that better input guarantees correct output.

The Fallacy

"If we give the AI better documentation, up-to-date APIs, and more context, it won't hallucinate."

Why it's tempting

You've seen the problem firsthand. You ask the AI to call an API. It uses a deprecated endpoint. You ask it to implement a library function. It invents a method that doesn't exist. You ask it to follow your team's coding conventions. It generates code that looks like it read the conventions from 2019.

The diagnosis seems obvious: the AI doesn't have the right information. The fix seems obvious: give it the right information. RAG retrieves relevant documents before the AI generates. Context Hub tools fetch the latest API documentation. System prompts inject your team's conventions. The context window fills with the information the AI needs.

And it works. The AI generates code that uses the current API. It follows the latest conventions. The hallucination rate drops measurably. The investment in context infrastructure pays off.

So what's the fallacy?

Why it's wrong

Better context reduces hallucination. It does not eliminate it. Production failures live in the gap between reduced and eliminated.

Three reasons context can't close the gap completely:

Reason 1: The AI can ignore the context

Retrieval-Augmented Generation retrieves documents and places them in the context window. The model is SUPPOSED to use them. But the model can — and does — override the retrieved context with patterns from its training data when the training data feels more natural to the generation process.

Liu et al. (2023) documented this as the "lost in the middle" problem: LLM performance degrades significantly when relevant information is placed in the middle of a long context window. Performance is highest at the beginning or end, and drops in between. This alone proves that "more context" is not a linear path to "more correctness."

But the problem is deeper than positional bias. Research into parametric versus non-parametric memory (Longpre et al., 2021; Neeman et al., 2022) shows that when an LLM's internal training weights (parametric) conflict with the provided RAG context (non-parametric), the model frequently defaults to its training — especially when the training data was highly reinforced. The model's internal weights represent billions of patterns. The context window represents a few thousand tokens. When the two conflict, the model doesn't reliably choose the context.

You can test this yourself: provide a document that explicitly contradicts the model's training data. Ask a question about it. The model will sometimes answer from the document and sometimes answer from its training. The percentage depends on the model, the prompt, and the specific conflict. It's never 100% from the document. The model isn't reading and following instructions. It's predicting the most likely next token given ALL its inputs — training weights and context window combined.

Reason 2: Context is necessary, verification is separate

Context answers: "What information should the AI use?"
Verification answers: "Did the AI use the information correctly?"

These are different questions with different mechanisms. Having the right ingredients doesn't guarantee the dish is correct. A chef with perfect ingredients can still combine them wrong, overcook the protein, or plate the wrong dish entirely. The ingredients are necessary. The taste test is still required.

Modern benchmarks confirm this gap. The Retrieval-Augmented Generation Benchmark (RGB) shows that models often fail to reason correctly over retrieved documents even when the retrieval is 100% accurate. Having the right documents in the context window and using them correctly are independent capabilities — and current models fail at the second even when the first is perfect.

In engineering terms:

Context pipeline:       Ensures the AI HAS the right information
                        → vector database, retrieval, ranking, injection
                        → measured by: retrieval accuracy, relevance scores

Verification pipeline:  Ensures the AI USED the information correctly
                        → specification checks, contract tests, property verification
                        → measured by: output correctness against declared properties

Most teams invest heavily in the first pipeline and have nothing for the second. They measure retrieval quality — "did we give the AI the right documents?" — but not output correctness — "did the output satisfy the properties it should satisfy?"

This is like a restaurant that invests millions in sourcing the finest ingredients, tracks every supplier relationship, monitors freshness down to the hour — but has no quality check on the finished dish. The ingredients are world-class. The plate that reaches the customer might still have no flavor or taste.

Reason 3: Context doesn't cover properties

Even with perfect context — every document retrieved, every API current, every convention injected — the AI has no mechanism to enforce PROPERTIES that span the generated output.

"No function in this module may call an external service without a timeout." That's a property. It's not in any API documentation. It's not in any retrieved document. It's an architectural decision the team made. The AI has no way to know about it from context alone, because it was never written as a retrievable document — it lives in the team's shared understanding.

"This data pipeline must preserve message ordering." That's a constraint. It's implicit in the architecture. Even if someone wrote it down, RAG would have to retrieve THAT SPECIFIC document for THAT SPECIFIC generation. The probability of the right document being retrieved for the right generation at the right time decreases as the number of such properties grows.

Context can provide facts. It can't provide the complete set of properties that must hold across every generated artifact. Properties are exhaustive ("ALL functions must have timeouts"). Context is sampled ("HERE are some relevant documents").

There's a deeper structural reason RAG misses properties. RAG retrieves by semantic similarity — vector math that finds documents "close to" the query in meaning space. A security invariant ("never log PII") is semantically FAR from a function that processes user data. They aren't similar in the vector space. The RAG system never retrieves the security rule for that specific generation — because security constraints and implementation code don't look alike in embeddings. Facts live near the code that uses them. Properties live in a different semantic neighborhood entirely.

Mishra et al. (2022) demonstrated a related failure: LLMs struggle significantly with negation and constraints in instructions. "Never log PII" is a negative property. Even when the constraint is explicitly in the context, the model's training data is composed almost entirely of positive examples ("do Y"), not negative ones ("never do X"). The context provides the constraint. The model's architecture biases against following it.

The boom

Month 1-3: The context investment. The team builds a sophisticated RAG pipeline. Vector database, document preprocessing, chunk optimization, relevance scoring. The AI generates better code. Hallucination rate drops from 15% to 5%. The team publishes an internal blog post celebrating the improvement.

Month 4-6: The 5% that matters. The 5% residual hallucination isn't random. It's concentrated in the hard cases — the ones where the context conflicts with the training data, the ones where the property isn't in any retrievable document, the ones where the AI needs to reason about composition rather than pattern-match individual facts. These are disproportionately the cases that cause production incidents.

Month 7-9: The false confidence incident. A developer generates code for a new feature. RAG retrieves the correct API documentation. The AI uses the correct API endpoint. The code compiles. The tests pass. But the code violates an architectural constraint — it makes a synchronous call to an external service without a timeout, inside a transaction. Nothing in the retrieved context mentioned the timeout requirement. The constraint was in the team's architecture decision records, which were never indexed in the vector database. The service hangs. The database connection pool exhausts. The outage lasts four hours.

Here's the cruel irony: better context made the hallucination MORE dangerous, not less. The code used the correct API. It followed the correct conventions. It looked MORE correct than code generated without RAG — which made everyone trust it more. Better context doesn't stop the AI from generating violations. It gives the AI better facts to wrap the violation in. The plausibility goes up. The scrutiny goes down. The damage goes up.

Post-mortem: "We had the right context. The AI used the right API. But the AI didn't know about the timeout constraint because we didn't retrieve it. And we didn't retrieve it because we didn't index it. And we didn't index it because we didn't know we needed to."

The post-mortem reveals the structural gap: context improves what the AI generates FROM. It doesn't check what the AI generates AGAINST. The retrieved documents were correct. The generated code was plausible. The property that was violated was never in the context pipeline — and no verification pipeline existed to catch it.

Month 10, the response: The team indexes their ADRs in the vector database. They add more documents to the RAG pipeline. The context gets better. The next incident is a different property that wasn't indexed. The whack-a-mole cycle begins — each incident reveals a property that should have been in the context but wasn't. The team can never index EVERYTHING because they don't have a complete list of everything that matters.

The resolution: verify the output, not just the input

Context and verification are complementary. Both are necessary. Neither is sufficient alone.

INPUT QUALITY (context):
    "Give the AI the right information"
    → RAG, Context Hub, up-to-date docs, system prompts
    → Reduces hallucination
    → Measured by: retrieval accuracy

OUTPUT CORRECTNESS (verification):
    "Check that the AI used the information correctly"
    → Specification gates, contract tests, property checks
    → Catches violations regardless of context quality
    → Measured by: properties satisfied per change

The team that invests only in context is optimizing input quality and hoping output correctness follows. Sometimes it does. The 5% where it doesn't is where the incidents live.

The team that adds verification checks the output REGARDLESS of how it was generated. Whether the AI had perfect context or no context at all, the verification catches violations — because the verification checks PROPERTIES, not INPUTS.

The kitchen analogy, completed

Context = Ingredients
    Better ingredients → better chance of a good dish
    But: a chef can still combine them wrong

Verification = Taste test
    Check the finished dish against the standard
    Catches errors regardless of ingredient quality

Great restaurants have BOTH:
    World-class sourcing AND quality checks on every plate

Great engineering has BOTH:
    RAG pipeline AND specification verification on every change

Investing in RAG without investing in verification is like a restaurant that sources the finest ingredients but has no chef tasting the dish before it reaches the customer. Most dishes will be fine. The ones that aren't will be the ones the customer remembers.

What the context pipeline can't replace

Five categories of problems that better context cannot solve:

1. Compositional correctness. The AI generates Function A correctly (right context retrieved) and Function B correctly (right context retrieved). The composition of A and B violates a cross-cutting property. No individual document in the RAG pipeline covers the composition — because the property emerges from the interaction, not from any single component.

2. Architectural constraints. "All inter-service calls must use gRPC with deadline propagation." This is an organizational decision, not a fact in a document. Even if indexed, RAG must retrieve it for EVERY generation that involves a service call. One missed retrieval = one violation.

3. Negative properties. "This function must NEVER log PII." Context tells the AI what TO do. Negative properties tell it what NOT to do. The AI can follow the positive context perfectly and still violate a negative property that wasn't in the retrieved documents — and even when the constraint is retrieved, the model's training bias toward positive examples means it struggles to follow negative instructions reliably (Mishra et al., 2022).

4. Implicit conventions. "Error handling in this codebase uses result types, not exceptions." The convention is embedded in 10,000 existing functions. RAG might retrieve a few examples. The AI might still generate exceptions because its training data favors exceptions. The context helps. It doesn't guarantee.

5. Mathematical properties. "The sum of all line items must equal the invoice total." This is arithmetic. The AI can have perfect context about the invoice schema and still generate code where floating-point arithmetic introduces a rounding error. The property requires verification, not context.

What you can do this week

1. List five properties that must hold in your system. Not facts the AI needs to know — PROPERTIES the output must satisfy. "All endpoints require authentication." "No database query uses string concatenation for parameters." "All monetary amounts use integer cents, not floating-point dollars." These are your verification targets.

2. Check: are any of these in your RAG pipeline? Probably not. Properties live in people's heads, in architecture decision records, in tribal knowledge. They're not the kind of documents teams typically index for retrieval. This gap is your vulnerability.

3. Add one property as a mechanical check. Not in the RAG pipeline — in the CI pipeline. A check that runs on every change, regardless of what context the AI had. The property either holds or it doesn't. The check is the verification layer that context can't provide.

4. Audit your RAG pipeline against your properties. Pick 10 recent successful retrievals — cases where RAG provided the right documents and the AI generated correct-looking code. For each one, ask: "If the AI had used this context perfectly but ignored our internal timeout policy (or our PII logging rule, or our authentication requirement), would anything have caught it?" If the answer is no for even one, you have a context-only architecture. The context is doing its job. The verification layer doesn't exist yet.

Your context pipeline is probably good. Keep investing in it. Better retrieval means fewer hallucinations. But add the verification pipeline alongside it — because the hallucinations that survive good context are the ones that cause the most damage. They're the subtle ones. The plausible ones. The ones that pass every test because nobody wrote a test for that specific property. And they're the ones that a mechanical check catches on every change, every time, regardless of what was or wasn't in the context window.

Next in the series: **Fallacy #6 — "AI-Generated Code Is an Asset."* Why every line of generated code is a liability, why the right unit of progress isn't code volume, and what Unix figured out about composable functions 50 years ago.*

The Fallacies of GenAI Development: eight assumptions every team is making. Each one leads to an architectural failure. Each one has already been solved.

References

Liu, N.F., et al. (2023). "Lost in the Middle: How Language Models Use Long Contexts." Transactions of the ACL.
Longpre, S., et al. (2021). "Entity-Based Knowledge Conflicts in Question Answering." EMNLP 2021.
Mishra, S., et al. (2022). "Cross-Task Generalization via Natural Language Crowdsourcing Instructions." ACL 2022.
Neeman, E., et al. (2022). "DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering." ACL 2023.

Fallacies of GenAI Development #4: Dropping Human Review Removes the Bottleneck

Bala Paranj — Tue, 02 Jun 2026 10:18:31 +0000

This is the fourth in a series of eight posts on the false assumptions teams make when building with generative AI. Fallacy #1 covered why faster generation doesn't mean faster engineering. Fallacy #2 covered why plausible isn't correct. Fallacy #3 covered why AI can't reliably verify AI. This post covers what happens when the review gate is removed entirely.

The Fallacy

"Human code review is the bottleneck. If we drop it, the pipeline moves at AI speed."

Why it's tempting

The math is simple and the frustration is real. AI generates a PR in 3 minutes. Human review takes 3 hours. The human is 60x slower than the machine. If you have five developers each generating AI-assisted PRs, one tech lead reviewing them becomes the constraint on the entire team's output. The pipeline stalls at the review stage.

The trending solution: drop the review. Let AI write 100% of the code. Trust the tests. Ship faster. Prominent voices in the industry advocate this explicitly — if the review is the bottleneck, remove it. The reviewer's time is better spent on higher-level work.

The logic is compelling until you ask one question: what replaced the review?

Research backs the concern. Perry et al.'s 2023 Stanford study ("Do Users Write More Insecure Code with AI Assistants?") proved that developers using AI assistants wrote more insecure code but were more likely to believe it was secure. The human "reviewer bottleneck" isn't just a speed problem — it's a cognitive failure caused by a false sense of security. The faster the AI generates, the more confident the developer becomes, and the less they scrutinize.

Why it's wrong

A gate exists for a reason. It catches things. When you remove a gate, the things it was catching start reaching production. The question isn't "is the review slow?" The question is "what was the review catching that nothing else catches?"

Human code review catches at least three categories of problems that no other part of the pipeline addresses:

1. Properties nobody wrote tests for. Tests verify what someone THOUGHT to verify. Review catches what nobody thought to test — an architectural violation, a security assumption, a performance implication, a business logic error that doesn't have a test case because nobody anticipated it.

2. Compositional correctness. Tests verify individual components. Review is often the only place someone looks at how components interact — does this change break the implicit contract between module A and module B? Does this new endpoint introduce a dependency cycle? Does this database migration interact badly with the concurrent migration from the other team?

3. Design coherence. Tests verify behavior. Review verifies intent — is this the right approach? Does this change align with the architecture? Are we building the right thing, or just building a thing that passes tests? This is judgment, not verification. But it's judgment that prevents the codebase from drifting into incoherence over hundreds of changes.

Drop the review and all three categories reach production unchecked. Not immediately — the tests still catch the easy bugs. But the hard problems — the architectural drift, the composition errors, the untested security assumptions — accumulate invisibly.

Manny Lehman proved this in 1980. His Second Law of Software Evolution states explicitly: "As an evolving program is continually changed, its complexity, reflecting deteriorating structure, increases unless work is done to maintain or reduce it." Code review is the primary mechanism for that active work. Removing it doesn't pause the entropy — it confirms that the entropy will accumulate until the system becomes unmanageable. Dropping review makes architectural collapse a mathematical certainty, not a risk.

Rigby and Bird's 2013 empirical study of thousands of code reviews at Microsoft and in open-source projects ("Convergent Software Peer Review Practices") found that the primary value of review isn't bug-finding. It is knowledge transfer, design improvement, and maintaining team-wide standards. Drop the review and you don't just miss bugs — you kill the Theory Building (Naur, 1985) and the Conceptual Integrity (Brooks, 1975) of the system.

The boom

Month 1-3: The velocity spike. PRs merge without waiting for reviewers. Feature delivery accelerates visibly. The team ships more in three months than the previous six. Leadership celebrates. Metrics look great.

Month 4-6: The silent accumulation. Each AI-generated PR that skipped review carried a small number of decisions nobody examined. A variable naming convention drifted. An error handling strategy diverged between services. A retry policy was implemented differently in three modules. None triggers a test failure. Each is a micro-crack in the architecture.

Month 7-9: The first incident nobody can diagnose. A production failure in a code path nobody understands. The developer on call opens the file. It was AI-generated. It was never reviewed. Nobody built a mental model of how it works. The developer reads the code — it looks correct. The bug is in the interaction between this function and another function in a different service, also AI-generated, also never reviewed. Debugging takes three days. Writing the code took three minutes.

Month 10-12: The architecture change that can't be made. The team needs to refactor a core module. The module has been modified by AI agents dozens of times since anyone reviewed it. The tests pass. The code reads well. But nobody knows WHY it's structured the way it is. Nobody knows which behaviors are intentional and which are accidental artifacts of AI generation. The team is afraid to change it. The code that was built in months can't be safely modified in months.

The deeper damage: ownership loss. When no human reviews the code, no human owns the code. The team discovers they aren't software engineers anymore — they're prompt operators, powerless to fix a system they didn't build and don't understand. The prompts generated the code. The code runs the business. Nobody in between can explain how.

Sarkar et al.'s 2024 study on developer experience with AI coding assistants ("What is it like to use a generative AI coding assistant?") found exactly this pattern: AI-assisted coding leads to "shallow understanding." Developers focus on the immediate fix and fail to build the deep mental model of how the change affects the rest of the system. If no human reviewed the code, the cognitive debt is at its maximum when the incident occurs.

This is the cognitive debt trajectory from Fallacy #1, now at the code level. The review was the only mechanism that built human understanding of AI-generated changes. Without it, the understanding was never formed. The debt compounds silently until it's called in — always during an incident, always at the worst possible time.

Three models of review

The industry is debating between two models. There are actually three.

Model 1 — Human reviews everything:
    AI generates code → Human reads every line → Ship

    ✓ Accurate (human judgment catches subtle issues)
    ✗ Slow (human is the bottleneck at 60x slower than AI)

    This is what teams had. It doesn't scale.

Model 2 — Nobody reviews:
    AI generates code → Tests pass → Ship

    ✓ Fast (pipeline moves at AI speed)
    ✗ Unsafe (properties nobody tested reach production unchecked)

    This is what teams are moving toward. It breaks at Month 7.

    Note: "AI code review" feels like Model 1 but is actually Model 2
    with a false sense of security. As Fallacy #3 showed, the AI
    reviewer has the same failure modes as the AI generator. The
    safety profile is Model 2. The confidence level is Model 1.
    That mismatch is where the damage compounds.

Model 3 — Specification gate reviews:
    AI generates code → Specification gate verifies → Ship
    Human reviews SPECIFICATIONS, not code

    ✓ Fast (gate operates at machine speed)
    ✓ Accurate (properties verified mechanically, exhaustively)
    ✓ Sustainable (human effort scales with specifications, not code volume)

    This is what every safety-critical domain converged on.

Model 2 looks like an upgrade from Model 1. It's actually a downgrade — it removed the safety mechanism without replacing it. The team traded accuracy for speed, calling it optimization.

Model 3 is the actual resolution. It doesn't compromise. It separates the two requirements:

ACCURACY operates on specifications (small, stable, human-authored, reviewed deliberately)
SPEED operates on code verification (fast, exhaustive, mechanical, every change)

Specification doesn't mean a 100-page requirements document. It means any machine-verifiable artifact of intent that already exists in your codebase: a TypeScript interface, an OpenAPI definition, a Protocol Buffer schema, a SQL migration, a Semgrep rule, a database constraint. Small. Stable. Human-authored. Machine-enforced.

The human reviews the rules of the game. The machine reviews every move to ensure the rules weren't broken.

How every safety-critical domain got here

No safety-critical domain uses Model 1 or Model 2. Every one uses Model 3. They arrived there independently when their version of "AI-speed generation" exceeded human review capacity:

Aviation: Jet engines got fast enough that pilots couldn't react to every condition. Model 1 (pilot monitors everything) was too slow. Model 2 (remove the pilot) was too dangerous. Model 3: fly-by-wire envelope protection. The pilot reviews the FLIGHT PLAN (specification). The computer enforces the FLIGHT ENVELOPE (mechanical gate). The pilot can't stall the aircraft even if they try — the specification gate overrides the input. This isn't ad hoc — DO-178C (the standard for flight software certification) requires that requirements (specifications) be reviewed by humans for intent, while code is verified against those requirements using deterministic tools: structural coverage analysis, data coupling analysis, formal methods. Humans never review every line of flight code. They review the specification of what the code must do, and machines verify every line against it.

Nuclear operations: Reactor dynamics happen faster than operators can track every parameter. Model 1 (operator monitors all parameters) was too slow. Model 2 (remove operator oversight) was too dangerous. Model 3: automated protection systems. The operator reviews the PROCEDURES (specification). The interlocks enforce PARAMETER LIMITS (mechanical gate). The reactor scrams automatically if parameters exceed the envelope — regardless of operator input.

Financial trading: Algorithmic execution happens in microseconds. Model 1 (human reviews every trade) was too slow. Model 2 (no review) caused flash crashes. Model 3: pre-trade risk checks. The risk manager reviews the RISK LIMITS (specification). The system enforces them on EVERY TRADE (mechanical gate). No order that violates the limits reaches the exchange — regardless of what the algorithm generated.

Google monorepo: 2 billion lines of code. Model 1 (human reviews every change to every dependency) was too slow. Model 2 (merge without review) would break the monorepo. Model 3: automated testing + API contract enforcement. The team reviews INTERFACE CONTRACTS (specification). CI enforces them on EVERY CHANGE (mechanical gate). A large-scale change touching millions of lines merges — because every affected test passes mechanically. As Winters, Manshreck, and Wright document in Software Engineering at Google (2020), the code review chapter makes this explicit: "mechanical checks" (linters, tests) are automated, but "design and intent" review is the gate that keeps the system coherent. Even the world's largest codebase doesn't drop review — it moves review to the highest level of abstraction.

The pattern repeats: when generation speed exceeds human review capacity, the review doesn't disappear. It splits into two activities at two speeds. The human does the slow, high-judgment work (reviewing specifications). The machine does the fast, exhaustive work (verifying code against specifications).

The TRIZ contradiction that forces Model 3

This isn't a preference. It's a resolution of a physical contradiction.

The review must be ACCURATE — it requires human domain expertise to judge whether the code is correct, whether the architecture is sound, whether the security properties hold.

The review must be FAST — human review speed is the bottleneck that prevents the team from capturing AI's productivity gains.

Same system. Opposite requirements. TRIZ's separation principle: if a system must be simultaneously A and not-A, separate the contradiction across different artifacts.

ACCURATE (human speed):
    Human authors specification    → small, slow, requires expertise
    Human reviews specification    → infrequent, high-judgment

FAST (machine speed):
    Machine verifies code          → instant, exhaustive, every change
    Machine blocks violations      → deterministic, no judgment needed

Both requirements satisfied. Neither compromised. The accuracy lives in the specification review. The speed lives in the mechanical verification. Different artifacts, different speeds, no contradiction.

What you can do this week

If you've already dropped human review (Model 2):

1. Identify what the review was catching. Look at your last 20 review comments before you dropped the process. Categorize: how many were about properties (should always be true), how many about composition (interaction between components), how many about design (is this the right approach)? The specification gate replaces the property-category comments.

2. Convert one review comment into a specification. "This endpoint must always require authentication" — that's a specification. Add it as a CI check. Mechanical. Deterministic. Every PR. You just restored one piece of what the review was providing, at machine speed.

3. Keep a "review debt" log. Every time an incident occurs in code that was never reviewed, log it. Track the category. After a quarter, the log tells you exactly which specifications you need. The incidents write the specification backlog for you.

If you still have human review (Model 1):

1. Identify the bottleneck reviews. Which reviews take the longest? Which ones block the most PRs? These are the candidates for Model 3 — convert the reviewer's judgment into specifications and enforce them mechanically.

2. Convert one slow review into a fast gate. The reviewer who checks "does this PR conform to our API contract" — replace that with a contract test. The reviewer who checks "does this migration have a rollback" — replace that with a CI check for rollback scripts. Each conversion speeds up the pipeline without removing the safety mechanism.

3. Measure reviewer time on specifications vs. code. If the reviewer spends 80% of their time checking properties (mechanical work) and 20% on design judgment (human work), there's a 4x speedup available by moving the mechanical work to CI. The reviewer focuses on the 20% that requires human judgment. The machine handles the 80% that doesn't.

The bottleneck is real. The review is slow. But removing the review is removing the brakes. The resolution is to move the review to the right level — specifications for humans, code verification for machines — and let each operate at its natural speed.

Next in the series: **Fallacy #5 — "Better Context Prevents Hallucination."* Why improving input quality doesn't guarantee output correctness, and why verification must check the output — not just improve the input.*

The Fallacies of GenAI Development: eight assumptions every team is making. Each one leads to an architectural failure. Each one has already been solved.

References

Lehman, M.M. (1980). "Programs, Life Cycles, and Laws of Software Evolution." Proceedings of the IEEE, 68(9), 1060–1076. Second Law: complexity increases unless active work is done to reduce it.
Rigby, P.C. & Bird, C. (2013). "Convergent Contemporary Software Peer Review Practices." ESEC/FSE 2013. Empirical study: review's primary value is knowledge transfer and design improvement, not bug-finding.
Perry, N. et al. (2023). "Do Users Write More Insecure Code with AI Assistants?" IEEE S&P 2023. Stanford study: AI-assisted developers write more insecure code while believing it is more secure.
Sarkar, A. et al. (2024). "What is it like to use a generative AI coding assistant? An interpretative phenomenological analysis." Microsoft Research. AI coding leads to shallow understanding and missed system-level effects.
DO-178C (2011). "Software Considerations in Airborne Systems and Equipment Certification." RTCA/EUROCAE. Aviation software standard: human-reviewed requirements, machine-verified code.
Winters, T., Manshreck, T. & Wright, H. (2020). Software Engineering at Google. O'Reilly. Chapter on code review: mechanical checks automated, design and intent review is the coherence gate.
Naur, P. (1985). "Programming as Theory Building." Microprocessing and Microprogramming, 15(5), 253–261.
Brooks, F.P. (1975). The Mythical Man-Month. Addison-Wesley. Conceptual integrity as the most important consideration in system design.

Fallacies of GenAI Development #3: You Can Verify AI Output With Another AI

Bala Paranj — Mon, 01 Jun 2026 11:17:30 +0000

This is the third in a series of eight posts on the false assumptions teams make when building with generative AI. Fallacy #1 covered why faster generation doesn't mean faster engineering. Fallacy #2 covered why plausible isn't correct. This post covers why using one AI to check another doesn't solve the problem — it doubles it.

The Fallacy

"If the AI makes mistakes, use another AI to check its work."

Huang et al. (ICLR 2024) showed that LLMs cannot reliably self-correct their reasoning without external feedback, and in some cases self-correction makes the output worse. LLM-as-judge is a special case of this: the same class of system evaluating its own output using the same reasoning that produced the errors. Formal verifiers, schema validators, and dissimilar reasoning engines provide the external feedback the paper says is required.

Why it's tempting

The logic feels airtight. You wouldn't trust a single developer to ship code without review. So you add a reviewer. In AI systems, the reviewer is another AI — a guardrail, a grader, an LLM-as-judge. The first AI generates. The second AI checks. Two opinions are better than one.

The industry has formalized this into named patterns:

Guardrails on input: An LLM checks whether the user's prompt is safe before the main LLM processes it.
Guardrails on output: An LLM checks whether the main LLM's response is appropriate before the user sees it.
LLM-as-judge: A grader LLM scores the quality of the main LLM's output against a rubric.
AI code review: An LLM reviews AI-generated code for bugs, security issues, and style.

Each pattern adds a layer of verification. Each layer is another LLM call. The architecture looks like defense in depth.

Why it's wrong

The verifier has the same failure modes as the thing it's verifying.

An LLM checking another LLM's output for hallucination can itself hallucinate. An LLM checking for prompt injection can itself be prompt-injected. An LLM reviewing code for security vulnerabilities can miss the same subtle patterns the generating LLM introduced — because both are doing the same thing: pattern matching on text.

This is not defense in depth. Defense in depth requires each layer to have DIFFERENT failure modes. A firewall and an intrusion detection system provide defense in depth because they fail differently. The firewall fails on novel protocols, the IDS fails on encrypted payloads. Neither failure mode overlaps with the other.

Two LLMs have OVERLAPPING failure modes. Both hallucinate. Both are gullible to adversarial inputs. Both miss mathematical properties that are invisible in the text. Both confuse plausibility with correctness. Adding a second LLM doesn't eliminate the failure mode. It adds another instance of it.

The math of stacked probabilities

If your generating LLM is 95% reliable and your guardrail LLM is 95% reliable, the combined reliability is NOT 99.75% (as it would be with independent failure modes). It's somewhere closer to 95% — because the failure modes are correlated. The cases where the generator fails are disproportionately the cases where the guardrail also fails, because both struggle with the same category of hard inputs.

The inputs that fool one LLM are often the inputs that fool the other. A policy that's mathematically equivalent to Principal: * through complex condition logic fools the generator (it doesn't understand the math) and fools the guardrail (it also doesn't understand the math). A prompt injection disguised as a legitimate system instruction bypasses the input guardrail for the same reason it would bypass the main model — both process it as plausible text.

The prompt injection example

This makes the problem concrete. The Fowler/Subramaniam GenAI patterns article recommends guardrails on input to prevent prompt injection. The architecture:

User prompt → Guardrail LLM ("is this safe?") → Main LLM → Response

The guardrail LLM reads the user's prompt and decides whether it contains an injection attempt. But the guardrail is ALSO an LLM processing text. A sufficiently clever injection that reads as legitimate to one LLM will read as legitimate to the other — because both are doing the same kind of text pattern matching.

Now consider the alternative architecture. Instead of a free-text prompt surface, expose typed RPC methods:

Agent calls: search(query="s3 buckets", scope="production")
Agent calls: verify(property="no_public_access", resource="arn:aws:s3:::data")
Agent calls: compliance(framework="hipaa", account="prod-account")

There's no prompt to inject into. The agent calls typed methods that accept typed parameters and return structured data. The guard isn't another LLM, it's the type system. The attack surface doesn't exist because the architecture doesn't have a free-text channel.

Prompt injection isn't mitigated. It's structurally unreachable. The architecture doesn't have the surface for the attack to exist.

The pattern across nine Fowler mitigations

This isn't just about prompt injection. Every reactive GenAI pattern shares the same structural limitation:

Reactive pattern	What it does	Why it doesn't converge on reliability
RAG	Provides context to reduce hallucination	The LLM can still ignore the context
Input guardrails	LLM checks if the input is safe	The guardrail LLM has the same gullibility
Output guardrails	LLM checks if the output is appropriate	The guardrail LLM has the same blind spots
LLM-as-judge	LLM scores output quality	The judge LLM has the same hallucination risk
Query rewriting	LLM improves the user's query	The rewriting LLM can make the query worse
Reranking	LLM rescores retrieved documents	The reranker LLM has the same relevance errors
Fine-tuning	Retrain the model on domain data	The fine-tuned model still hallucinates
Evals	LLM scores output against rubrics	The eval LLM has probabilistic accuracy
AI code review	LLM reviews code for bugs	The reviewer LLM misses the same edge cases

Each pattern wraps a non-deterministic system with another non-deterministic layer. Each layer adds cost and latency. None makes the underlying unreliability structurally unreachable. They reduce the probability of failure. They don't eliminate the failure mode.

The boom

Teams that rely on AI-to-verify-AI hit a specific wall: the recursive hallucination.

Month 1: The team deploys an LLM guardrail to check AI-generated IAM policies before deployment. The guardrail catches obvious issues — wildcard principals, missing conditions. Leadership is satisfied. We have AI reviewing AI.

Month 3: A subtle policy change passes both the generator and the guardrail. The policy's condition blocks, evaluated together, are mathematically equivalent to Principal: *. Neither LLM understands the math. Both pattern-matched the text. The text looks restrictive. The math isn't.

Month 4: The incident. A public access path exists through the policy composition. Data is exposed.

Month 4, the post-mortem: "Why did the guardrail miss this?" Because the guardrail reads policy TEXT the same way the generator writes policy TEXT. It checked whether the text LOOKED like it restricted access. It didn't check whether the LOGIC restricted access. You didn't have two independent checks. You had two mirrors reflecting the same error back at each other.

Month 5, the response: The team adds a THIRD LLM to review the guardrail's decisions. The recursive hallucination deepens. The cost triples. The latency triples. The correlated failure mode remains. The next incident will be a different policy composition that all three LLMs approve because all three pattern-match text, not logic.

The recursive hallucination is the AI verification equivalent of an infinite loop. Each iteration adds cost without adding reliability, because each layer fails on the same category of inputs.

What deterministic verification looks like

The alternative isn't better AI verification. It's verification that doesn't use AI at all for the properties that can be checked mechanically.

Probabilistic (LLM-as-judge):
    "Does this code violate the security policy?"
    → Depends on the grader's interpretation
    → Run it twice, might get different answers
    → Misses mathematical equivalences invisible in text

Deterministic (mechanical check):
    "Does this configuration expose a public endpoint without authentication?"
    → Yes or no. Every time. Same input, same output.
    → Computed over the actual state, not over the text description
    → Catches mathematical equivalences because it evaluates logic, not text

Deterministic verification has specific tools at each level of sophistication:

Type checking. The compiler verifies type contracts. If the AI generates code that violates a type signature, the compiler catches it — instantly, deterministically, with zero LLM calls. This is verification at the speed of code generation with zero probability of false negatives for the properties types can express.

Contract testing. Tools like Pact and Dredd verify that an API implementation matches its OpenAPI specification. Every endpoint, every field, every response code — checked mechanically against the declared contract. The specification is the source of truth. The implementation either matches or it doesn't.

Property-based testing. Tools like QuickCheck and Hypothesis verify that a property holds across thousands of randomly generated inputs. Not "does this specific input produce this specific output?" but "does this property hold for ALL inputs the tool can generate?" One level of abstraction higher than example-based testing. One level closer to proof.

Static analysis. Tools like Semgrep, SonarQube, and Checkov check structural properties of the code without running it. "Does any code path reach a database query with unsanitized user input?" "Does this Terraform plan create a public S3 bucket?" Checked across the entire codebase or infrastructure definition, mechanically, on every commit.

Formal verification. Tools like Z3, Dafny, and Lean prove properties mathematically across ALL possible inputs. "Does there exist any request that this policy allows from a public principal?" If the solver says UNSAT, no such request exists — proved, not tested. AWS uses this (Zelkova) to verify IAM policies billions of times per day.

Each tool is deterministic. Each produces the same answer for the same input every time. Each catches a category of errors that LLM-as-judge cannot — because the errors are mathematical, not textual.

When to use which

Not every property can be verified deterministically. The decision tree:

Can the property be expressed as a type constraint?
    → YES: Use the type system. Cheapest. Fastest. Already deployed.

Can the property be expressed as a contract (API spec, schema)?
    → YES: Use contract testing. Mechanical. Deterministic.

Can the property be expressed as "for all inputs, X holds"?
    → YES: Use property-based testing. Thousands of random inputs.
           Or formal verification for mathematical proof.

Can the property be expressed as a structural pattern in code?
    → YES: Use static analysis. No runtime needed. Every commit.

Is the property about tone, style, user experience, or subjective quality?
    → YES: Use LLM-as-judge. This is the ONLY category where
           probabilistic verification is appropriate — because the
           property itself is subjective.

LLM-as-judge is appropriate for subjective properties. It's inappropriate for properties that have deterministic answers. Using an LLM to check whether code violates a type contract is like using a language model to check whether 2 + 2 = 4. You could. But the calculator is faster, cheaper, and never wrong.

The principle: match the verifier to the property

The resolution isn't never use AI for verification. It's use the right verification method for each property.

Subjective properties    → LLM-as-judge (probabilistic, appropriate)
    "Is this response helpful?"
    "Is this code well-styled?"
    "Is this documentation clear?"

Structural properties    → Static analysis (deterministic)
    "Does this code path sanitize user input?"
    "Does this function handle the error case?"

Contract properties      → Contract testing (deterministic)
    "Does this API match its specification?"
    "Does this response include all required fields?"

Universal properties     → Property-based testing / formal verification (deterministic)
    "Can any unauthorized principal access this resource?"
    "Does the balance ever go negative for valid transactions?"
    "Does there exist any input that causes this function to hang?"

Most properties in a security-critical, financially-critical, or data-integrity-critical system are NOT subjective. They have deterministic answers. Verifying them with an LLM is using the wrong tool — because a better tool exists for that specific category.

The teams that emerge from the trough of disillusionment fastest will be the ones that stop using AI to verify everything and start matching each property to the verification method that works for it. Subjective? LLM. Everything else? Mechanical.

What you can do this week

1. Audit your current AI verification stack. List every place you use an LLM to check another LLM's output. For each one, ask: is the property being checked subjective or deterministic? If it's deterministic, you're using the wrong tool.

2. Replace one probabilistic check with a deterministic one. If you have an LLM reviewing code for "does it match the API spec" — replace it with a contract test. If you have an LLM checking "is the output valid JSON" — replace it with a schema validator. One replacement. This week.

3. Measure the difference. How many violations did the deterministic check catch that the LLM missed? How much faster does it run? How much cheaper is it? The numbers will make the case for replacing the rest.

The LLM is a remarkable tool. Use it where it's good at: subjective judgment, creative generation, natural language understanding. Don't use it as a calculator. Don't use it as a type checker. Don't use it as a theorem prover. Those tools already exist. They're faster, cheaper, and they never hallucinate.

Next in the series: **Fallacy #4 — "Dropping Human Review Removes the Bottleneck."* Why removing a gate without replacing it isn't optimization — it's removing the brakes. Three models of review, and why the third one is the only one that works at AI speed.*

The Fallacies of GenAI Development: eight assumptions every team is making. Each one leads to an architectural failure. Each one has already been solved.

The Industry Needs an Open Reasoning Spec. Seven Papers Explain What Goes In It.

Bala Paranj — Sun, 31 May 2026 16:44:20 +0000

The API ecosystem had a coordination problem. Every API was described differently — prose documentation, custom schemas, tribal knowledge. Then a standard format emerged. One format. Every tool reads it. Code generators, documentation engines, test frameworks, mock servers, SDK builders — all consume the same specification. The ecosystem unified around one artifact.

The AI era has the same coordination problem — but for a different artifact. Not "how does this API behave?" but "what properties must this system preserve?" "What behavioral contracts must hold?" "Why was this boundary placed here?" "What invariants must every change respect?"

Seven foundational CS papers — Parnas, Naur, Brooks, Knuth, Dijkstra, Liskov, Lehman — converge on the same conclusion: the value of software isn't in the code. It's in the properties, contracts, boundaries, and rationale that make the code safe to modify. AI generates the code. Nothing standardizes the properties. The code is generated at scale. The properties are scattered across READMEs, ADRs, Slack threads, and people's heads.

The industry needs an Open Reasoning Spec — a machine-readable standard for properties, contracts, and rationale that AI agents consume, enforcement engines verify, and humans read.

What exists today (and why it's insufficient)

Several standards describe parts of the problem:

Standard        What it describes              What's missing
────────        ─────────────────              ──────────────
OpenAPI         API shapes (endpoints, types)  Behavioral contracts (idempotency,
                                               ordering, invariants)

JSON Schema     Data shapes (fields, types)    Semantic constraints (this field
                                               means admin access, not just boolean)

OSCAL           Compliance controls            Mechanical predicates (how to CHECK
                                               the control, not just describe it)

STIX            Threat intelligence            Property definitions (what must be
                                               TRUE, not just what threats exist)

OPA/Rego        Policy rules                   Rationale (WHY this policy exists,
                                               what decision it implements)

CEL             Predicate expressions          Context (what the predicate means
                                               in domain terms, who validated it)

ADRs            Decision rationale             Enforcement link (the ADR says WHY
                                               but nothing connects it to a CHECK)

Each standard covers one layer. None connects them. The API shape (OpenAPI) doesn't link to the behavioral contract (Liskov). The compliance control (OSCAL) doesn't link to the mechanical predicate (CEL). The decision rationale (ADR) doesn't link to the enforcement gate (CI check). The standards exist in silos — exactly like API descriptions before OpenAPI.

What the seven papers require

Each foundational paper identifies a specific element that must be in the specification:

Paper	What must be specified	Current gap
Parnas (1972)	Module boundaries — what's inside, what's outside, what the interface promises	Boundaries exist in code (packages, modules) but aren't declared in a consumable spec
Naur (1985)	Domain theory — how the software maps to the real world	Lives in people's heads. No format captures it.
Brooks (1986)	Essential complexity — which parts are irreducibly hard	Not distinguished from accidental complexity in any specification
Knuth (1984)	Narrative — WHY each decision was made	ADRs exist but aren't machine-readable or linked to enforcement
Dijkstra (1972)	Verifiable properties — what must be TRUE, stated simply enough to prove	Properties scattered across tests, types, linter rules, and comments
Liskov (1994)	Behavioral contracts — promises beyond the type signature	No standard format. Lives in comments, test names, tribal knowledge.
Lehman (1980)	Structural maintenance rules — what's allowed to grow, what must be consolidated	No standard. Implicit in code review culture.

No existing standard captures all seven. Most capture zero or one. The reasoning specification must capture all seven in one artifact — because they're interdependent. The boundary (Parnas) is meaningless without the contract it enforces (Liskov). The contract is unverifiable without the property definition (Dijkstra). The property is unmaintainable without the rationale (Knuth). The rationale is lost without the boundary that preserves it (Parnas).

The shape of an Open Reasoning Spec

An Open Reasoning Spec (working name — the community will name it) is a machine-readable document that describes what must be TRUE about a system, WHY it must be true, and HOW to check it. Three sections, mapping to the three layers:

Section 1: Boundaries (Layer 1 — Parnas)

boundaries:
  - id: boundary.auth.module
    description: "Authentication module — all auth logic lives here"
    interface:
      exports:
        - function: Authenticate(credentials) → (session, error)
        - function: ValidateSession(token) → (claims, error)
      imports_allowed:
        - crypto/
        - database/sessions
      imports_forbidden:
        - business/  # auth must not depend on business logic
        - api/       # auth must not depend on API layer
    enforcement:
      tool: depguard
      config_path: .depguard.yml
      ci_gate: true

The boundary declares: what the module exports (the interface), what it's allowed to import (dependencies), and what it's forbidden to import (architectural boundaries). The enforcement section links the boundary to the tool that checks it. An agent reading this spec KNOWS the boundary before generating code. A CI gate reading this spec ENFORCES the boundary on every commit.

Section 2: Properties and contracts (Layer 2 — Dijkstra + Liskov)

properties:
  - id: prop.auth.idempotent
    description: "Authenticate is idempotent — same credentials produce same session"
    scope: boundary.auth.module
    type: behavioral_contract
    predicate: |
      for all (c: Credentials):
        Authenticate(c) == Authenticate(c)
    verification:
      method: property_test
      tool: gopter
      test_path: auth/auth_property_test.go
    mitre_attack: T1078.004  # relevant threat technique

  - id: prop.iam.no_admin_escalation
    description: "No principal can escalate to admin through any role chain"
    scope: global
    type: safety_invariant
    predicate: |
      for all (p: Principal, r: Role):
        can_assume(p, r) AND is_admin(r) implies is_admin(p)
    verification:
      method: cel_predicate
      expression: "properties.identity.escalation.*.present != true"
      tool: stave
    validated_against:
      - vendor: datadog
        lab: iam-002-to-admin
        result: detected
      - vendor: bishopfox
        lab: privesc4-CreateAccessKey
        result: detected

Each property declares: what must be true (the predicate), how to check it (the verification method and tool), what threat it addresses (MITRE mapping), and what independent oracle validated it (lab traceability). The predicate is machine-readable — an enforcement engine can evaluate it. The description is human-readable — a developer can understand WHY.

Section 3: Rationale (Layer 3 — Knuth + Naur)

rationale:
  - id: rationale.auth.idempotency
    decision: "Authenticate must be idempotent"
    date: 2024-03-15
    author: engineering-team
    context: |
      Incident INC-2024-03-12: retry middleware called Authenticate
      twice for the same request. The non-idempotent implementation
      created two sessions. The user received two session cookies.
      Subsequent requests alternated between sessions, producing
      intermittent authorization failures.
    consequences:
      - Sessions are keyed by credential hash, not by call sequence
      - Duplicate calls return the existing session, not a new one
    properties_enforced:
      - prop.auth.idempotent
    supersedes: null
    temporal_validity: active

The rationale records WHY the decision was made (the incident), WHAT consequences it has (implementation constraints), WHICH properties enforce it (linked to Section 2), and WHETHER it's still active (temporal validity). When the decision is superseded, the temporal_validity changes and the linked properties are flagged for review.

This IS the context graph ThoughtWorks describes — but with enforcement links. The rationale connects to the property. The property connects to the verification. The verification connects to the CI gate. The chain is: WHY → WHAT → HOW → CHECKED.

Adapted, not copied

The seven papers identified the properties. The Open Reasoning Spec doesn't implement them as the authors originally described — because the authors wrote for human developers, not for AI-assisted development. Each idea was adapted for the new context:

Paper	Original idea	Problem with original in AI era	Adaptation in the spec
Knuth (1984)	Rationale as comments interwoven with code	AI overwrites comments on regeneration. Comments go stale silently. Comments are coupled to code that changes.	Rationale in a SEPARATE artifact (Section 3) that the AI reads but doesn't modify. Linked to enforcement so staleness is detectable.
Dijkstra (1972)	Simplicity for human reasoning about correctness	AI generates code too complex for humans to reason about. Simplicity can't be enforced by asking.	Properties as MECHANICAL CHECKS (Section 2) that verify correctness without requiring the human to reason about the implementation. The check replaces the reasoning.
Liskov (1994)	Behavioral contracts as documentation and convention	Documentation is ignored. Convention is violated by AI that doesn't know the convention.	Contracts as EXECUTABLE PREDICATES (Section 2) that CI evaluates on every change. The contract is enforced, not documented.
Parnas (1972)	Module boundaries as design decisions in the developer's head	AI doesn't hold design decisions. The developer who prompted the AI didn't make the boundary decision.	Boundaries as DECLARED STRUCTURE (Section 1) with enforcement links. The boundary is in the spec, not in someone's head.
Naur (1985)	Theory built through the act of writing code	The act of writing was replaced by prompting. Theory is never built.	Theory EXTERNALIZED into the spec — boundaries + properties + rationale = the theory in durable form. The spec IS the theory, persisted independently of the people who authored it.
Lehman (1980)	Structural maintenance through human discipline	AI generates faster than humans can maintain. Discipline doesn't scale.	Maintenance rules as AUTOMATED CHECKS — deletion metrics, dependency constraints, complexity thresholds in Section 2. The maintenance is mechanical, not disciplinary.
Brooks (1986)	Essential complexity recognized through the struggle of implementation	AI eliminates the struggle. Essential complexity is hidden under perfect-looking output.	Essential complexity NAMED EXPLICITLY as properties in Section 2. The spec forces the author to state what's hard — which properties are domain-specific, which constraints are irreducible. Naming it makes it visible.

The common adaptation across all seven: what the original paper located in the developer's MIND (theory, understanding, discipline, convention, struggle), the spec locates in a DURABLE, ENFORCEABLE ARTIFACT. The mind is fragile — people leave, forget, get overridden by AI. The artifact persists, is version-controlled, is mechanically enforced, and survives personnel changes.

The specific innovation for the AI era: enforcement replaces documentation. Every previous attempt to capture properties, contracts, and rationale produced DOCUMENTATION — comments, ADRs, convention guides, design docs. Documentation goes stale because nothing checks whether it's still accurate. The Open Reasoning Spec connects every rationale entry to a mechanical check. When the check fails, the rationale is reviewed. When the rationale is superseded, the check is updated. The connection between WHY and WHAT is maintained mechanically, not by human memory.

This is the difference between "writing better comments" (Knuth's original) and "declaring enforceable properties with linked rationale" (the adaptation). The idea is the same — capture WHY. The mechanism is different — enforce it instead of commenting it. The enforcement is what makes it survive in a world where AI generates and regenerates code continuously.

What this enables

For AI agents

The agent reads the spec before generating code. It knows: the auth module can't import from business/. The Authenticate function must be idempotent. The boundary is enforced by depguard. The agent generates code WITHIN these constraints — not because it was prompted to, but because the constraints are declared in a machine-readable format the agent consumes as context.

The agent doesn't need the full codebase in context. It needs the reasoning spec. The spec is smaller than the code (hundreds of lines vs. thousands). It contains the information the seven papers say matters: boundaries, properties, contracts, rationale. The code is the implementation. The spec is the theory.

For enforcement engines

The CI gate reads the spec and runs every verification: depguard checks boundary imports, gopter runs property tests, Stave evaluates CEL predicates, Z3 checks satisfiability. Each property has a declared verification method. The gate runs them all. The exit code is pass or fail. No human triage. No alert queue.

When a property fails, the gate produces: WHICH property failed, WHAT the predicate says, WHY the property exists (linked rationale), and WHAT to do about it (remediation from the property definition). The developer sees: "prop.auth.idempotent FAILED because Authenticate created a new session for duplicate credentials. This property exists because of INC-2024-03-12 (see rationale). Fix: key sessions by credential hash."

For humans

The developer reads the spec and understands the system at the boundary level — Parnas's information hiding. They don't need to read the implementation. They read the interfaces, the properties, and the rationale. The spec IS Naur's theory, externalized into a durable artifact. The theory survives personnel changes because it's in the spec, not in someone's head.

For interoperability

Different tools consume different sections. The linter reads boundaries. The property tester reads contracts. The compliance engine reads properties with MITRE mappings. The AI agent reads all three. The Open Reasoning Spec is the lingua franca between tools — the same way API description standards unified the API tooling ecosystem.

Tool                Consumes from spec
────                ──────────────────
depguard            boundaries.imports_forbidden
gopter              properties.predicate (property tests)
Stave               properties.predicate (CEL expressions)
Z3                  properties.predicate (SMT-LIB export)
Soufflé             properties.predicate (Datalog rules)
CI gate             all verification sections
AI coding agent     boundaries + properties + rationale
Human developer     boundaries + rationale (readable sections)
Compliance auditor  properties + validated_against (evidence)

One spec. Ten consumers. Each reads the section it needs. The ecosystem builds around the spec the same way the API ecosystem built around OpenAPI.

What must be true about the standard

For an Open Reasoning Spec to succeed, it needs five properties that successful standards demonstrate:

1. Machine-readable AND human-readable. YAML or JSON with clear field names. The developer reads it. The CI gate parses it. Same document.

2. Incrementally adoptable. A team can start with one boundary definition and one property. They don't need to spec the entire system on day one. Each section is independently useful.

3. Tool-agnostic. The spec describes WHAT to check, not WHICH TOOL checks it. The verification section names the tool but the predicate is tool-independent. A property that says "this function is idempotent" can be checked by gopter, QuickCheck, Hypothesis, or any property testing framework.

4. Extensible. New property types, new verification methods, new rationale formats can be added without breaking existing specs. The same way mature standards add capabilities without breaking existing consumers.

5. Versionable. The spec lives in version control alongside the code. Changes to the spec are reviewed in PRs. The spec evolves with the system. Temporal validity on rationale entries handles superseded decisions.

The precedent

API description standards didn't invent API descriptions. Multiple competing formats existed. The standard that succeeded unified the fragments into one format that the ecosystem adopted. The key wasn't inventing something new — it was standardizing what already existed into one interoperable format.

The reasoning spec fragments already exist: ADRs describe rationale. Type signatures describe boundaries. Property tests describe contracts. Linter configs describe structural rules. OSCAL describes compliance controls. CEL/Rego/OPA describe policy predicates.

The standard that unifies them — connecting rationale to property to enforcement to verification in one document — doesn't exist yet. The seven foundational papers describe what it must contain. The three-layer model describes its structure. The ecosystem (AI agents, enforcement engines, compliance tools, human developers) describes its consumers.

Every standard emerges when the coordination problem becomes acute enough. The reasoning coordination problem is becoming acute now — because AI generates code at scale without consuming the properties, contracts, and rationale that make the code safe to modify. The Open Reasoning Spec will emerge when the cost of NOT having it exceeds the cost of creating it. That point is approaching fast.

From Wild West to engineering discipline

Right now AI-assisted development is in its Wild West phase. Every team picks different tools. Every tool solves a different fragment. No shared vocabulary describes what "correct AI-assisted output" means. Companies evaluate tools by token throughput and generation speed — the equivalent of evaluating a construction company by how fast they pour concrete, not by whether the building stands.

The waste is measurable. Tokens spent generating code that fails review. Tokens spent regenerating code that fails CI. Tokens spent debugging AI-generated code that nobody understands. Tokens spent on feedback flywheels that improve surface quality without mechanical verification. The industry is spending billions on code generation and near-zero on code verification. The ratio is inverted.

An Open Reasoning Spec changes this in three ways:

1. Companies know what to evaluate. Instead of "which AI coding tool generates code fastest?" the question becomes "which tool consumes the reasoning spec and produces output that satisfies the declared properties?" The spec is the evaluation criteria. A tool that generates code that violates the spec's properties is failing — regardless of how fast it generates. A tool that generates less code but satisfies every property is succeeding. The spec makes "correct" measurable.

2. Tooling and ecosystem unify around the standard. Today: fragmented solutions that don't interoperate. The linter checks boundaries but doesn't know about behavioral contracts. The property tester checks contracts but doesn't know about rationale. The AI agent generates code but doesn't know about either. With a standard: every tool reads the same spec. The linter reads boundaries from Section 1. The property tester reads contracts from Section 2. The AI agent reads all three sections before generating. The compliance auditor reads the validated_against entries. One artifact coordinates the entire toolchain. The ecosystem builds around it instead of fragmenting without it.

3. Enforcement becomes automated. The spec declares properties. The CI gate checks them. The check is mechanical — deterministic, independent of the model, runs at machine speed. Every token spent generating code that violates a declared property is a wasted token. The enforcement gate catches it before the code reaches review, before it reaches staging, before it reaches production. The waste moves from "discovered in production" to "caught at generation time." The cost of a violation drops from "incident + remediation + post-mortem" to "regenerate."

This is how every engineering discipline matured. Civil engineering had its Wild West — buildings collapsed, bridges failed, each builder used different standards. Then building codes emerged. The codes declared properties: load-bearing capacity, wind resistance, seismic tolerance. The inspection process verified the properties mechanically. Builders who met the code shipped. Builders who didn't, didn't. The code was the standard. The inspection was the enforcement. The ecosystem (architects, engineers, inspectors, material suppliers) unified around the code.

Software engineering is the last major engineering discipline without a standard for declaring and enforcing the properties that matter. The Open Reasoning Spec is that standard. The seven foundational papers describe what it must contain. The three-layer model describes its structure. Six safety-critical domains proved the resolution works. The only question is when the industry adopts it — not whether.

The Wild West ends when enforcement becomes automated. Not when AI gets better at generating code. Not when feedback flywheels improve prompt quality. Not when context graphs capture more rationale. Enforcement. Automated. Mechanical. Deterministic. Independent of the model. That is the transition from code generation to engineering.

Stave implements a domain-specific version of the Open Reasoning Spec for cloud security: the observation contract (JSON Schema Draft 2020-12) defines boundaries, the control YAML defines properties with CEL predicates, the defect/infection/failure model captures rationale, and the SIR export enables multi-engine verification (SMT-LIB for Z3, Datalog for Soufflé, facts for Prolog). Every property is lab-validated against independent expert oracles. The format is specific to cloud security. The structure — boundaries + properties + rationale, machine-readable, incrementally adoptable, tool-agnostic — is the structure the open standard needs. Apache 2.0.

A 1972 Paper Predicted the AI Coding Crisis. Nobody Listened.

Bala Paranj — Sun, 31 May 2026 11:40:30 +0000

There's a paper from 1972 that describes — with precision — the crisis AI coding agents are creating right now.

David Parnas published "On the Criteria To Be Used in Decomposing Systems into Modules" when software ran on mainframes and a large program was 10,000 lines. The paper argued that the conventional way of breaking systems apart (by execution steps: input → process → output) was wrong, and that systems should instead be decomposed by design decisions that are likely to change.

The reason was human cognition. Not compiler efficiency. Not runtime performance. The hard bottleneck of software engineering, Parnas argued, is the developer's ability to understand the system well enough to change it safely. Modular boundaries exist to protect human comprehension. Everything else is secondary.

Fifty-three years later, AI coding agents generate complex logic 5-7x faster than humans can comprehend it. Developers accept code through what can only be called cognitive surrender — they review the output, the tests pass, and they merge it without building a mental model of how or why it works.

Parnas diagnosed this exact failure mode half a century before it existed. And his cure still works.

The Firehose Effect

Here's what happens in practice. You prompt an AI agent to implement a retry mechanism with exponential backoff and circuit breaking. The agent returns 300 lines of well-structured code: a RetryPolicy struct, a CircuitBreaker with state management, a BackoffCalculator with jitter, integration tests, and error classification logic that distinguishes transient from terminal failures.

The code works. The tests pass. You read it, nod at the structure, and merge it.

Three weeks later, you need to change the retry behavior for one specific endpoint. You open the file. You can't remember how the circuit breaker interacts with the backoff calculator. You're not sure whether changing the max-retries constant affects the circuit breaker's failure threshold. You don't know if the error classifier's transient category includes timeouts or only HTTP 5xx responses.

You wrote none of this code. You reviewed it once. You understood it well enough to approve it. You didn't understand it well enough to change it safely.

That gap — between understanding enough to approve and understanding enough to change — is cognitive debt. It lives in your head, not in the code. The code is fine. Your mental model of the code is incomplete. And it became incomplete the moment the AI bypassed the cognitive struggle that would have built it.

Parnas's term for the cause: the system violated information hiding. Not because the code was poorly written, but because the module boundaries weren't designed to protect your comprehension. The circuit breaker and the backoff calculator and the error classifier are all exposed to you simultaneously. When you need to change one, you have to reason about all three. The code didn't hide its design decisions. It dumped them on you. That's the firehose.

Parnas's argument (it's not about code organization)

The 1972 paper is widely cited and widely misunderstood. Most developers think Parnas argued for "clear module boundaries" — a code organization principle. He argued for something deeper: that the criterion for where to place module boundaries should be based on which design decisions are likely to change.

The conventional approach decomposed systems by processing steps: an input module, a processing module, an output module. Each step becomes a module. Parnas showed this was wrong — not because it's messy, but because it forces developers to understand design decisions that cross module boundaries. When the input format changes, the processing module has to change too, because it knows about the input format. The design decision (input format) leaked across the boundary. The developer changing the input format now has to understand two modules instead of one.

Parnas's alternative: hide each volatile design decision inside a single module. The input format lives in one module. The processing algorithm lives in another. The output format lives in a third. When the input format changes, only the input module changes. The developer only needs to understand one module. The interface between modules is stable. The implementation behind the interface is hidden.

The key word is hidden. Not "encapsulated." Not "abstracted." Hidden. The developer using the module cannot see the implementation. Cannot depend on it. Cannot be confused by it. The design decision is invisible outside its module. The developer's cognitive load is bounded by the interface, not by the implementation.

Why AI violates information hiding by default

AI coding agents don't think in Parnas modules. They think in tasks.

"Implement a retry mechanism with circuit breaking" is a task. The AI produces a solution to the task — often a good one. But the solution is organized around the task's execution logic, not around which design decisions should be hidden from the developer.

The retry policy, the circuit breaker state machine, and the error classification logic all land in the same module (or in tightly coupled modules) because they all serve the same task. The AI doesn't ask "which of these design decisions might change independently?" It asks "what code solves the prompt?"

The result is exactly the decomposition Parnas warned against: organized by execution steps (receive request → classify error → calculate backoff → check circuit state → retry or abort) rather than by design decisions (error classification policy, backoff strategy, circuit threshold). The three design decisions are tangled together. Changing one requires understanding all three.

This isn't an AI quality problem. A skilled human developer writing under time pressure produces the same tangle. The difference is that the human developer at least struggled through the implementation and built a partial mental model of the tangled decisions. The AI generates the tangle instantly, and the developer never builds the model at all.

When the AI generates at this speed across dozens of modules, the effect compounds. Each module has leaking design decisions. Each leaked decision is a piece of implementation the developer must understand to make safe changes. Multiply this across an AI-accelerated codebase and you get what the cognitive debt analysis calls the firehose effect — the volume of exposed design decisions exceeds human cognitive capacity.

Cognitive firewalls

Parnas's solution, applied to AI-generated code, creates what amounts to a cognitive firewall:

[ AI Agent generates complex implementation ]
                │
                ▼
┌──────────────────────────────────────┐
│      PARNAS MODULE INTERFACE         │  ← Developer reviews THIS
│  (what it provides, requires,        │
│   guarantees — the contract)         │
├──────────────────────────────────────┤
│  Hidden AI-Generated Internals       │  ← Hidden from working memory
│  (complex, fast, possibly opaque)    │
└──────────────────────────────────────┘

The developer doesn't need to understand the AI's internal algorithmic choices. They need to understand the interface: what the module provides, what it requires, what it guarantees, and what happens when those guarantees are violated.

For the retry example, the Parnas decomposition hides each design decision:

Module 1: ErrorClassifier
  Interface: classify(error) → transient | terminal
  Hidden: the classification rules, the regex patterns,
          the HTTP status code mappings

Module 2: BackoffStrategy  
  Interface: nextDelay(attempt) → duration
  Hidden: the backoff curve, the jitter algorithm,
          the maximum delay cap

Module 3: CircuitBreaker
  Interface: allow() → bool; record(outcome)
  Hidden: the state machine, the failure threshold,
          the recovery window, the half-open probe logic

Each interface is 1-2 lines. Each hidden implementation might be 50-100 lines of AI-generated code. The developer understands three interfaces (6 lines). The AI generated 150-300 lines behind them. The developer's cognitive load scales with the interfaces, not with the implementation.

When the developer needs to change the backoff strategy, they modify Module 2. They don't touch Module 1 or Module 3. They don't need to understand the circuit breaker's state machine to change the backoff curve. The design decisions are hidden. The cognitive firewall holds.

The criterion AI agents should follow

Parnas's criterion for decomposition — "what design decisions are likely to change?" — is the instruction that's missing from most AI coding workflows.

When a developer prompts "implement retry with circuit breaking," the AI should ask (or the developer should specify): "which design decisions in this implementation might change independently?"

The answer for the retry example:

The error classification rules will change when new error types are discovered
The backoff strategy will change when performance requirements shift
The circuit breaker thresholds will change when reliability targets change
The integration between them (retry loop orchestration) will change rarely

Each independently-changeable decision becomes a module. The integration becomes a thin orchestrator that calls the modules through their interfaces. The developer understands the orchestrator (10 lines) and the interfaces (6 lines). The implementations are hidden.

This is not new. Parnas published it in 1972. What's new is that AI makes the violation of this principle automatic and invisible. The AI generates monolithic implementations because monolithic implementations solve the prompt efficiently. The Parnas decomposition is a constraint the developer must impose on the AI's output — the AI won't impose it on itself.

Unix already proved this

Unix's design is Parnas applied at the operating system level. Every Unix tool hides its implementation behind a universal interface: stdin, stdout, exit codes.

grep doesn't expose its regex engine. sort doesn't expose its sorting algorithm. uniq doesn't expose its deduplication logic. Each tool is a Parnas module with a hidden implementation and a stable interface (text streams in, text streams out).

The result: a developer who has never read a line of grep's source code can compose grep | sort | uniq and predict the output with certainty. The cognitive load is bounded by the interface (text stream processing) not by the implementation (regex finite automata, merge sort, hash-based deduplication).

AI-generated code should compose the same way. Each module should have a stable interface that a developer can reason about without reading the implementation. The implementation can be AI-generated, opaque, and complex. The interface must be human-readable, stable and simple.

When the AI generates a module whose interface is as complex as its implementation, it has failed the Parnas test — regardless of whether the code works.

Interfaces evolve into contracts

Parnas described interfaces within a single program, in a single language. But the principle doesn't stop at the module boundary. The industry has been walking an interface evolution for decades — and each step applies the same information-hiding insight at a larger scale.

Level 1: Within a module — language-level interfaces. Go's interface{}, Java's interface, TypeScript's type signatures. The developer defines the contract in the same language as the implementation. type ErrorClassifier interface { Classify(err error) ErrorKind }. The compiler enforces it. The implementation is hidden behind the type system. Every developer already knows this level.

Level 2: Between modules — package boundaries. Go's internal/ convention. Java's module system. Package-level exports vs. private internals. The developer sees the exported API. The unexported implementation is hidden by the language's visibility rules. Same Parnas principle, enforced by the language's module system rather than by discipline alone.

Level 3: Between systems — API contracts. OpenAPI, gRPC/Protocol Buffers, GraphQL schemas. The interface is no longer in the same language as the implementation. The API contract is expressed in a language-independent schema. A Python client and a Go server share the OpenAPI spec — neither needs to understand the other's implementation, or even what language it's written in. This was the first major jump: information hiding across language boundaries. The contract became a standalone artifact, separate from any implementation.

Level 4: Between organizations — standard formats. JSONL, SARIF, SMT-LIB, OCSF. The interface is no longer between two specific systems — it's a standard that any system can implement. SARIF doesn't belong to any security scanner; it's a format any scanner can emit and any dashboard can consume. The contract became universal. The implementation became fully interchangeable.

Level 5: Between human and machine — specifications. This is the level the cognitive debt crisis demands. The interface is no longer between two pieces of software. It's between a human's intent and a machine's execution. A CEL predicate, a reasoning spec YAML, a typed invariant — these are contracts between the developer who knows what must be true and the engine (or AI agent) that makes it true.

The evolution at each level:

Level	Scope	Developer reasons about	Interface construct	Examples
1	Within a program	A function — inputs, outputs, behavior	Language-level signatures	Go `interface{}`, Java `interface`, C function prototypes
2	Between modules	A module — responsibilities, boundaries	Package/module exports	Go `internal/`, Java 9 modules, C header files (`.h`)
3	Between systems	A system — capabilities, failure modes	API contracts, protocols	Unix pipes (stdin/stdout), RPC, REST/OpenAPI, gRPC/Protobuf
4	Between organizations	An ecosystem — interoperability, data exchange	Standard formats	SARIF, JSONL, EDI, SWIFT messages, HL7/FHIR, OCSF
5	Between intent and execution	Intent — what must always be true	Specifications, invariants	CEL predicates, reasoning specs, TLA+ properties

Each level raises the abstraction. At Level 1, the developer reasons about control flow and data types within a single program. At Level 3, Unix proved that processes don't need to share memory or language — grep | sort | uniq composes through stdin/stdout, the universal system-level interface. At Level 4, organizations that have never communicated can exchange data because SARIF and HL7 define the shape independently of any implementation. At Level 5, the developer reasons about what must be true, regardless of how any program, module, or system implements it.

At each level, the same thing happens: the interface enables reasoning without understanding the implementation. At Level 1, you reason about the function without reading its body. At Level 3, you reason about the service without knowing its language. At Level 5, you reason about the system's safety without reading any code at all.

But something deeper shifts alongside the interface: what the developer understands changes.

At Level 1, the developer understands a function — its signature, its preconditions, its return type. The unit of understanding is a single operation.

At Level 2, the developer understands a module — its exported API, its responsibilities, its boundaries. The unit of understanding is a coherent set of operations.

At Level 3, the developer understands a system's capability — what the service does, not how it does it. "This service classifies errors" is a capability. The developer integrates with the capability through a contract (OpenAPI spec), not by reading source code. Integration becomes easier precisely because the contract specifies the capability at a level above the implementation.

At Level 4, the developer understands a data shape — what the standard format carries, what fields mean, what consuming tools expect. SARIF findings have a ruleId, a level, a message. Any tool that emits this shape integrates with any dashboard that reads it. Understanding the shape is understanding the integration.

At Level 5, the developer understands an invariant — what must always be true about the system. Not how it's enforced. Not what code checks it. What property holds. "No unauthorized principal can read sensitive data through any path in the access graph" is a statement about the system's meaning, not its mechanism.

Notice what happens to durability as you climb. Understanding a function's implementation is fragile — it changes with every refactor, every AI regeneration, every optimization. Understanding a module's API is more durable — APIs change less often than implementations. Understanding a system's capability is more durable still — capabilities persist across rewrites. Understanding an invariant is the most durable of all — "no unauthorized access to sensitive data" doesn't change when you switch languages, rewrite the backend, or replace the team.

Cognitive debt compounds fastest at the lowest levels of understanding, because that's where change is fastest and understanding is most fragile. Level 1 understanding (function internals) erodes the moment AI regenerates the function. Level 5 understanding (safety invariants) persists across every regeneration, because the invariant describes intent, not implementation.

This is why the specification-first model resolves cognitive debt: it moves the developer's understanding to the level where it's most durable. You stop understanding the code (Level 1-2, fragile, erodes with every AI-generated change) and start understanding the invariants (Level 5, durable, persists across implementations). The code becomes disposable. The understanding doesn't.

The industry already walked Levels 1 through 4. Every developer has written a Go interface (Level 1), consumed an OpenAPI spec (Level 3), and emitted SARIF or JSONL (Level 4). The patterns are familiar. The cognitive skill — reasoning about the contract rather than the implementation — is already trained.

Level 5 is the same skill applied one level higher. Instead of "I don't need to understand the HTTP handler's implementation, I need to understand the OpenAPI spec," it's "I don't need to understand the AI-generated codebase, I need to understand the safety invariants." The reasoning is the same. The level of abstraction shifts. The cognitive load drops — because the specification is always smaller than the implementation, at every level, by definition. That's Parnas's insight, scaled to its logical conclusion.

What makes Level 5 urgent now is AI. At Levels 1-4, humans wrote the implementations behind the interfaces. The implementations were slow to produce, so cognitive debt accumulated gradually. At Level 5, AI generates the implementations instantly. The implementations are vast, fast-changing, and opaque. The only thing that keeps the developer's understanding intact is the specification — the Level 5 interface. Without it, the developer has no contract to reason against. They're back to reading AI-generated code line by line, which is Stage 2 feedback — the level Parnas proved was insufficient in 1972.

Technical debt vs. cognitive debt

The Parnas lens sharpens the distinction between technical debt and cognitive debt:

	Technical debt	Cognitive debt
Where it lives	In the repository — messy code, poor formatting, tight coupling	In human heads — eroded understanding of how and why the system works
How it's detected	Linters, static analysis, code smells, compiler warnings	Crises, outages, fear of changing code, "don't touch that module" folklore
Parnas diagnosis	Bad code structure that fails the modularity test — design decisions leaked across boundaries	Breakdown of the interface contract — the internals have overwhelmed the developer's ability to reason about the module
How it compounds	Linearly — each shortcut adds friction	Exponentially — each un-understood module makes adjacent modules harder to understand
How AI affects it	AI can reduce it — AI-generated code is often well-formatted and structurally clean	AI accelerates it — well-formatted code that nobody understands is still cognitive debt

The last row is the critical one. AI-generated code often has less technical debt than human-written code — better naming, consistent formatting, comprehensive error handling. And it simultaneously creates more cognitive debt — because the developer didn't build the mental model during writing.

Tackling technical debt (reformatting, renaming, extracting functions) doesn't reduce cognitive debt. The code looks better. The developer's understanding is unchanged. The cognitive debt measures something the linter can't see: does the developer understand this code well enough to change it safely?

Parnas's answer: they don't need to understand the code. They need to understand the interface. If the module boundary is drawn correctly (hiding volatile design decisions), the developer's understanding of the interface is sufficient for safe changes. The implementation is someone else's problem — whether someone else is another developer, another team, or an AI agent.

What this means for AI-assisted development

Three concrete practices follow from applying Parnas to AI-generated code:

Specify the module boundaries before prompting the AI. Don't prompt "implement retry with circuit breaking." Prompt: "implement three modules: an ErrorClassifier with interface classify(error) → transient | terminal, a BackoffStrategy with interface nextDelay(attempt) → duration, and a CircuitBreaker with interface allow() → bool, record(outcome). Each module hides its implementation. Then implement a RetryOrchestrator that calls them through their interfaces." The Parnas decomposition is the developer's job. The implementation is the AI's job.

Review interfaces, not implementations. When the AI returns the code, review the interfaces first. Are they stable? Do they hide the volatile design decisions? Can a developer who reads only the interfaces predict the module's behavior? If the interface exposes implementation details (a circuit breaker interface that reveals its internal state enum), send it back. The interface is the cognitive firewall. If the firewall leaks, the module fails regardless of how correct the implementation is.

Measure cognitive debt by interface complexity, not code complexity. A module with 500 lines of AI-generated implementation behind a 2-line interface has low cognitive debt. A module with 50 lines of AI-generated code and a 20-line interface has high cognitive debt — the developer must understand 20 interface elements to use it safely. The ratio of interface complexity to implementation complexity is the Parnas metric for cognitive exposure. When AI generates code, track the interface complexity, not the line count.

The 53-year head start

Parnas proved in 1972 that human cognitive capacity is the hard constraint of software engineering. Not compilation speed. Not runtime performance. Not deployment frequency. The developer's ability to understand the system well enough to change it safely — that's the bottleneck that everything else depends on.

For 53 years, this insight was treated as a code organization principle: define modules, use good interfaces, practice encapsulation. It was right but not urgent. Humans wrote code slowly enough that they built mental models as they went. The cognitive bottleneck was real but manageable.

AI made it urgent. The cost of writing code dropped to near zero. The cost of understanding code stayed constant. The gap between production speed and comprehension speed — the gap Parnas identified as the fundamental constraint — is now growing exponentially.

The cure is the same as Parnas prescribed: decompose by design decisions, hide implementations behind stable interfaces, bound the developer's cognitive load to the interface rather than the implementation. The AI generates the implementation. The developer owns the interface. The interface is the understanding.

Parnas gave us a 53-year head start on solving this crisis. We just didn't know we'd need it until the AI made the constraint visible.

This article is part of a series on cognitive debt in AI-assisted development. Related: "Six Contradictions Behind Cognitive Debt" (TRIZ analysis), "Six Industries That Already Solved Cognitive Debt" (cross-domain precedents), and "The Next Platform Won't Track Code. It'll Track Intent." (the specification-first platform).

Google Has 1,000 Platform Engineers Making Security Invisible. You Have Zero. Here's How Agents Close the Gap.

Bala Paranj — Sat, 30 May 2026 11:57:33 +0000

Google's internal infrastructure makes misconfiguration impossible. A platform SYNTHESIZES safe configurations from one-line declarations. Developers write service: order-processor. The platform produces IAM roles, network policies, TLS certificates, monitoring, secrets management — all from pre-approved templates.

It works. Near-zero misconfiguration incidents internally. The approach is PROVEN.

It cost Google a thousand-plus platform engineers, a decade of investment, and organizational authority most companies don't have. Spotify built Backstage. Netflix built the Paved Road. Shopify built Polaris. Each invested YEARS and TEAMS to reach the same outcome.

The rest of the industry without platform teams — gets told: be more careful.

There's an alternative to templates or more platform engineers. AGENTS executing reasoning engines against machine-verifiable contracts. The same security guarantees, without the platform team. Because the reasoning is independent of the cloud provider, it works on every cloud simultaneously.

Three eras of cloud security scaling

Era	Approach	How it scales	Who can afford it
Era 1	Manual audits, reviews, training	Hire more people (linear)	Anyone — but it doesn't work at scale
Era 2	Human-coded templates + internal platforms	Better platform teams (sub-linear)	Google, Netflix, Spotify, Shopify — top 5%
Era 3	Agent-driven reasoning against formal specs	Adding agents + reasoning engines (logarithmic)	Anyone with CI/CD

Era 1 is where 80% of organizations are today. More training and reviews. More "be more careful." Linear scaling: double the developers, double the security bugs, need double the security engineers.

Era 2 is where the Big Tech giants are. Templates, Golden Paths, Paved Roads. The platform team absorbs the complexity. Sub-linear scaling: more developers don't produce proportionally more bugs because the platform prevents the bugs structurally. It works. But it costs millions per year in platform engineering salaries.

Era 3 is what happens when you make the reasoning MACHINE-EXECUTABLE. Instead of human-coded templates that generate safe configurations, you have machine-verifiable contracts that PROVE configurations are safe. The agents don't generate code, they evaluate state against invariants using formal reasoning engines. Logarithmic scaling: adding reasoning capacity (more specs, more engines) covers exponentially more configurations.

How each Big Tech model maps — and where agents evolve it

Spotify's Golden Paths → Reasoning specs as logic gates

Spotify's model: Software Templates in Backstage. Developer clicks "Create Secure Storage." Backstage synthesizes the Terraform, IAM roles, and monitoring automatically. The right way is the easiest way.

The limitation: Templates are STATIC. Human-authored. Cover pre-approved patterns only. A developer building something custom that doesn't fit a template falls off the Golden Path and is on their own.

The agent evolution: Reasoning specs act as LOGIC GATES for any path — not just pre-approved templates. While Spotify's templates verify that the CHOSEN PATH is safe, agents executing reasoning specs verify that ANY CONFIGURATION meets the same invariants. Custom paths get the same safety guarantees as Golden Paths.

Netflix's Paved Road → Agents as GPS + safety inspector

Netflix's model: Freedom and Responsibility. Stay on the Paved Road (using ConsoleMe for IAM, standard deployment tools) and everything is automated and secure. Go off-road and you're responsible for your own security.

The limitation: Off-road is where innovation happens. And off-road is where breaches happen. The model accepts that custom work is inherently riskier.

The agent evolution: Agents executing reasoning engines (Z3 for satisfiability, Soufflé for reachability, Prolog for logic programs) provide Paved Road-level assurance EVEN FOR OFF-ROAD configurations. The agent doesn't care whether the configuration came from a template or from a developer's custom Terraform. It evaluates the STATE against invariants. Custom configurations get formally verified — not just template-checked.

Shopify's abstracted infrastructure → Machine-readable contracts as the API

Shopify's model: Minimize the surface area of choice. Developers interact with an internal platform that hides Kubernetes and cloud provider complexity. The platform synthesizes configuration based on application needs.

The limitation: The platform team must maintain the abstraction layer. Every new cloud feature requires platform-team work to incorporate. The abstraction lags the cloud.

The agent evolution: JSON schemas and published contracts ARE the abstraction layer — but machine-readable. The contracts define what safe state looks like for each asset type. Agents can consume these contracts directly, playing the role of the "Platform Team" without the platform team. New cloud features get covered by adding a schema and controls — not by rebuilding the abstraction layer.

Google's formal verification → The same rigor, externalized

Google's model: Internal formal verification of infrastructure invariants. The most rigorous approach — and the most expensive to staff.

The limitation: Requires formal-methods expertise that's rare and expensive. Google can hire it. Most organizations can't.

The agent evolution: The formal rigor lives in the REASONING ENGINES (Z3, Soufflé, Clingo, Prolog, PRISM) — not in the operator's head. The operator writes invariants as CEL predicates. The system exports standardized facts (JSONL, SMT-LIB). The reasoning engines consume the facts and produce formally grounded results. The operator gets Google-level formal verification without needing to write TLA+ or hire formal-methods PhDs.

The staffing math

The argument becomes undeniable:

Era 2 staffing (Big Tech IDP model):

Platform team:           5-20 engineers (ongoing)
Security architects:     2-5 (ongoing)
Template authors:        3-8 (ongoing)  
Cloud-specific experts:  1-3 per cloud provider
Total:                   15-40 engineers dedicated to the platform
Cost:                    $3M-$10M/year in salaries alone
Time to value:           12-24 months

Era 3 staffing (agent-driven reasoning):

Security architect:      1 (writes reasoning specs + catalog controls)
Infrastructure:          Existing CI/CD pipeline
Reasoning engines:       Open source (Z3, Soufflé, Clingo, Prolog)
Cloud-specific work:     Steampipe collectors (community-maintained)
Total:                   1 engineer + existing infrastructure
Cost:                    One salary + open-source tooling
Time to value:           Days to weeks

The shift from 15-40 engineers to 1 engineer isn't a quality trade-off. It's an ARCHITECTURAL trade-off. The platform-team model puts the intelligence in HUMAN-WRITTEN TEMPLATES. The agent-driven model puts the intelligence in MACHINE-EXECUTABLE REASONING SPECS. The specs are reusable, composable, and formally verifiable. The templates are bespoke, cloud-specific, and manually maintained.

Staffing Era 3: Era 3 still requires "Policy Governance." While you don't need 40 engineers to build the platform, you still need security stakeholders to define the invariants (e.g., "What counts as a 'Production' asset?").

The Agent definition: AI Agents (LLMs) and Reasoning Agents (Symbolic Logic) distinction: This article focuses on Symbolic AI (Z3, Prolog), which is much more reliable for security than Generative AI (LLMs). We are talking about reasoning engines over chatbots.

The multi-cloud problem — and why it kills most approaches

Here the architecture produces an advantage no cloud vendor can match.

The Vendor Lockin Problem

AWS Security Hub, Google Security Command Center, Azure Defender — each offers multi-cloud support. What they mean: "send all your Azure and GCP logs into OUR database."

The cloud provider wants to be the manager of managers. They use multi-cloud as a way to pull data INTO their ecosystem. They don't want to make it easy for you to leave. Their multi-cloud story is a lock-in strategy wearing an interoperability costume.

The black-box problem with security vendors

Wiz, Prisma Cloud, Orca — each is genuinely multi-cloud. Their agents scan AWS, Azure, and GCP. But the logic they use to determine risk is a proprietary black box.

If you want to change how a public bucket is defined differently for Azure vs AWS (because your organization has different policies per cloud), you wait for the vendor to update their product. You can't write your own formal proof. You can't inspect the logic. You can't extend it. You rent the verdict.

The universal translation layer

Because the architecture uses an intermediate representation — standardized facts in JSONL and SMT-LIB format — the reasoning is INDEPENDENT of the cloud provider.

A reasoning spec for transitive reachability is written ONCE. Because the input data is normalized into machine-verifiable contracts through vendor-neutral schemas, the same Z3 or Prolog code works for AWS, Azure, and GCP.

Traditional multi-cloud:
    Write "public bucket" rule for AWS    (AWS-specific syntax)
    Write "public bucket" rule for Azure  (Azure-specific syntax)  
    Write "public bucket" rule for GCP    (GCP-specific syntax)
    Maintain 3 versions. Test 3 versions. Debug 3 versions.

Agent-driven reasoning:
    Write reasoning spec ONCE
    Agents apply it to normalized facts from ANY cloud
    One spec. One test. One truth.

The staffing implication in a multi-cloud world:

Task	Traditional multi-cloud	Agent-driven reasoning
Hiring	Need an AWS expert, an Azure expert, and a GCP expert	Need ONE security architect who understands the contracts
Logic	Write rules 3 times in 3 syntaxes	Write the reasoning spec ONCE; agents apply to all clouds
Context	3 different dashboards, 3 different finding formats	One unified stream of machine-readable facts
New cloud	Hire another expert, write rules again	Map new cloud to existing schemas; specs work immediately

The Comparison

AWS Security Hub    = a FEATURE of AWS      (you're locked in)
Wiz                 = a SERVICE you rent     (you can't inspect or extend the logic)
Stave               = a PROTOCOL you own     (the reasoning is yours, runs anywhere)

A feature lives inside a vendor's ecosystem. A service lives inside a vendor's infrastructure. A protocol lives inside YOUR infrastructure — air-gapped, credential-free, extensible, inspectable.

No cloud vendor can offer provider-independent reasoning because their primary goal is selling their own compute and storage — not making you cloud-agnostic. No security SaaS can offer inspectable reasoning because their business model depends on the logic being proprietary. The architecture that's logic-first and cloud-second can only be built by someone whose business model DOESN'T depend on cloud lock-in or proprietary logic.

The democratization argument

Google, Spotify, Netflix, and Shopify discovered that human error is a scaling constant. Double the developers → double the security bugs. Unless you change the RELATIONSHIP between the developer and the infrastructure.

They changed the relationship through massive platform-team investment. Templates. Golden Paths. Paved Roads. Abstracted infrastructure. It worked. It cost millions per year. It's inaccessible to the 95%.

The agent-driven reasoning model changes the same relationship through a different mechanism: instead of human-authored templates that generate safe code, machine-executable specs that PROVE code is safe. The mechanism is different. The outcome is the same: developers can't produce unsafe configurations because the system catches them — whether the configuration came from a template or from scratch.

The secret sauce that let Big Tech scale without security collapse was never the TEMPLATES. It was the INVARIANTS — the knowledge of what safe means, expressed in a form the machine can evaluate. Templates are ONE way to express invariants. Reasoning specs against machine-verifiable contracts are ANOTHER. The second way doesn't require a platform team.

You can finally have the Google/Netflix security model without hiring 1,000 platform engineers. The invariants are in the catalog. The reasoning is in the engines. The evaluation is in the pipeline. The platform team is replaced by agents executing formal proofs against standardized facts.

That's not multi-cloud support. That's multi-cloud abstraction. Not a feature of one cloud. Not a service you rent. A protocol you own.

How this relates to existing compliance mods

The Era-3 agent model needs two things existing per-resource framework mods can't structurally provide: machine-verifiable compositional contracts (agents reason across resources, not within them) and an evaluation surface independent of the cloud-provider's SQL schema (so the agent reuses one reasoning vocabulary across AWS, GCP, Azure, K8s). turbot/steampipe-mod-aws-compliance ships ~540 controls across 16+ frameworks and is the right tool for "render me a CIS dashboard for the auditor" — its SQL is tied to live AWS APIs by design. Stave's CEL predicates + JSON-Schema-anchored snapshot + nine-engine export are the agent-consumable form: authorship-agnostic, provider-independent, composition-aware. Two surfaces, complementary jobs, both render in Powerpipe — see github.com/sufield/stave/blob/main/docs/comparison/aws-compliance-mod.md for the side-by-side.

Era 3 cloud security: agent-driven reasoning against machine-verifiable contracts. CEL predicates evaluated against air-gapped snapshots. Standardized facts (JSONL, SMT-LIB) exported for nine external reasoning engines (Z3, Soufflé, Clingo, Prolog, PRISM, and more). Provider-independent. Logic-first. Cloud-second. Stave, an open-source risk reasoning engine. The Google security model without the Google platform team. Try it: bash examples/demo-ai-security/run.sh

Six Domains Already Solved AI's Cognitive Debt Problem. Software Is the Last to Learn.

Bala Paranj — Fri, 29 May 2026 15:17:28 +0000

Nuclear submarine crews operate reactors they can't fully comprehend in real-time. Microprocessor engineers design chips with billions of transistors no single person understands. Air traffic controllers manage thousands of aircraft simultaneously. Medical specialists treat patients whose full condition exceeds any one doctor's knowledge. Legal systems maintain millions of laws no lawyer has read entirely. Google engineers work in a monorepo of 2 billion lines nobody holds in their head.

Each domain faced the same crisis software development is facing now: the system became too complex for human comprehension, and the consequences of misunderstanding became catastrophic. Each domain solved it. Each arrived at the same three-layer structure independently.

Software development is the last domain to hit this wall — because AI-assisted code generation is the forcing function that made the complexity exceed human comprehension in months rather than decades. The solution already exists. It's been proven across six domains. The only thing missing is the correct vocabulary to carry it over.

That vocabulary requires separating three debts that the industry currently conflates.

Three debts, three locations

Technical debt:
    Location:  The codebase
    What it is: Code that works but is structured poorly — 
                shortcuts, missing abstractions, duplicated logic
    Who named it: Ward Cunningham (1992)
    Key property: DELIBERATE — the team chose the shortcut
                  knowingly, intending to pay it back later

Cognitive debt:
    Location:  The developer's mind
    What it is: Loss of comprehension — the developer cannot
                reason about the system, cannot predict its behavior,
                cannot modify it safely
    Who named it: Margaret-Anne Storey, building on Peter Naur's
                  "Programming as Theory Building" (1985)
    Key property: The debt lives in PEOPLE, not in code.
                  The code can be well-structured. The developer still
                  doesn't understand it.

Intent debt:
    Location:  The system's enforcement layer
    What it is: The gap between what the system SHOULD preserve
                and what is actually expressed as an executable
                constraint that the system CAN enforce
    Who named it: Margaret-Anne Storey — formalized in "From Technical
                  Debt to Cognitive and Intent Debt: Rethinking Software
                  Health in the Age of AI" (arxiv 2603.22106, March 2026).
                  Storey defines intent debt as the absence of externalized
                  rationale that developers and AI agents need to work 
                  safely with code.
    Key property: The debt lives in the ABSENCE of a mechanism.
                  The intent exists (someone decided something)
                  but the system has no way to enforce it.

    Note: Storey's definition emphasizes externalized KNOWLEDGE
    broadly. This article narrows the focus to externalized
    EXECUTABLE CONSTRAINTS specifically — the subset of intent
    that can be mechanically enforced. The distinction matters:
    an ADR that explains WHY is externalized knowledge (reduces
    intent debt in Storey's framing). A CI check that ENFORCES
    the why is an executable constraint (reduces intent debt in
    this article's framing). Both are necessary. The executable
    constraint is what makes the intent durable at machine speed.

These are independent. You can have any combination of the three. A system can have zero technical debt (well-structured code), high cognitive debt (nobody understands it), and high intent debt (no constraints are expressed). Or well-structured code, full comprehension, but no mechanical enforcement. Each combination produces different failures and requires different solutions.

Why the distinction matters

When the terms are conflated, the solutions are wrong:

If you think cognitive debt IS intent debt:
    You try to fix comprehension by writing specifications.
    But specifications don't make the developer understand the code.
    They make the SYSTEM enforce constraints. The developer's mental
    model is still missing. The specification gate catches violations —
    but the developer can't debug, modify, or extend the code because
    they never built the theory of how it works.

If you think intent debt IS cognitive debt:
    You try to fix enforcement by improving developer understanding.
    Pair programming, code reviews, documentation. The developer
    understands the system — but the understanding lives in their head.
    When they leave, the intent leaves with them. When AI generates
    code at 3 AM, no human is reviewing. The understanding exists.
    The enforcement doesn't.

If you think either IS technical debt:
    You try to fix both by refactoring code.
    But the code might already be well-structured. Cognitive debt doesn't live
    in the code — it lives in the developer's mind. Intent debt doesn't
    live in the code — it lives in the absence of a constraint.
    Refactoring addresses neither.

Each debt has a different solution because each lives in a different place.

The Parnas root

David Parnas's 1972 paper "On the Criteria To Be Used in Decomposing Systems into Modules" is the common root of both cognitive debt and intent debt — though Parnas didn't use either term.

His insight: humans can't hold entire systems in their heads. His solution: information hiding. Each module exposes an interface (what it does) and hides its implementation (how it does it).

The module interface is a single artifact that addresses BOTH debts simultaneously:

As a comprehension mechanism (addresses cognitive debt): the developer only needs to understand what the module promises, not how it works internally. The interface is the boundary of comprehension. The implementation behind the boundary is irrelevant to anyone outside the module. This is how Boeing engineers work on a 787 — each engineer understands their subsystem's interface contracts, not the entire aircraft.

As an intent mechanism (addresses intent debt): the interface IS the declaration of what must be true at this boundary. The function signature, the API contract, the type constraint — each is an expressed intent. The system can enforce it. The compiler checks type signatures. The contract test checks API conformance. The database engine checks schema constraints.

Parnas didn't need to distinguish cognitive debt from intent debt because in 1972, one act produced both: the developer DESIGNED the module interface, which simultaneously built their mental model (comprehension) and declared the contract (intent). The act of writing code was the mechanism that produced both outputs.

What AI broke

AI-assisted development decoupled comprehension from intent expression. The Su-Field model from TRIZ makes the decoupling visible.

Before AI: one field produces two useful outputs

Su-Field Model — Manual Development

         F (writing code)
        / \
       /   \
      /     \
    S1       S2
 Developer   Codebase

 Writing code is the FIELD that connects the developer
 to the codebase. This single field produces TWO outputs:

 Output 1: Comprehension (cognitive debt ↓)
   The developer builds a mental model WHILE writing.
   Typing, debugging, rewriting — each forces internalization
   of the logic, the edge cases, the constraints.

 Output 2: Expressed intent (intent debt ↓)
   The code itself captures decisions. The function signature
   IS the interface contract. The module boundary IS the
   architectural intent. The test IS the specification.

 One field. Two outputs. Both debts managed by a single act.
 The system is COMPLETE — no missing interactions.

In TRIZ terms, this is a complete Su-Field system: two substances (developer, codebase) connected by one field (writing code) that produces the desired effects (comprehension + expressed intent). The system works because the field simultaneously acts on both the developer (building understanding) and the codebase (encoding intent).

With AI: the field changes, both outputs disappear

Su-Field Model — AI-Assisted Development

         F' (prompting)
        / \
       /   \
      /     \
    S1       S3
 Developer   AI Agent
                |
                | generates code
                ↓
               S2
            Codebase

 The field changed from WRITING CODE to PROMPTING.
 A new substance (S3: AI agent) was inserted between
 the developer and the codebase.

 Output 1: Comprehension → LOST
   The developer prompts. The AI writes. The developer
   never types the code, never debugs the logic, never
   struggles with the edge cases. The mental model is
   never formed. Cognitive debt accumulates.

 Output 2: Expressed intent → LOST
   The AI generates code from patterns, not from the
   developer's architectural decisions. No explicit
   interface contract was declared. No module boundary
   was designed. The AI's output captures PATTERNS
   from training data, not INTENT from the developer.
   Intent debt accumulates.

 The field (prompting) doesn't produce either output.
 The system is INCOMPLETE — both useful effects are missing.

In TRIZ terms, the field substitution (writing → prompting) eliminated both useful outputs. The system went from complete (one field, two outputs) to incomplete (one field, zero outputs). The inserted substance (AI agent) produces a new output (code, faster) but doesn't produce the two outputs the old field provided (comprehension, expressed intent).

The TRIZ resolution: add fields, don't restore the old one

TRIZ says: don't try to restore the original field (that would mean going back to manual coding). Instead, add new fields that independently produce the missing outputs.

Su-Field Model — Resolved System

    Read top to bottom. The AI generates code into the codebase.
    Two additional fields act on the same codebase independently.

                S1 Developer
                |
                | F' (prompting)
                ↓
                S3 AI Agent
                |
                | generates code
                ↓
                S2 Codebase
               / \
              /   \
             /     \
            /       \
           ↓         ↓
          F2          F3
    Parnas boundaries  Mechanical enforcement
           |           |
           ↓           ↓
      S1 Developer   S4 Specification gate
      COMPREHENDS    ENFORCES intent
      at interface   on every change
      level          mechanically
    (cognitive       (intent
     debt ↓)          debt ↓)

F'  produces code (fast, AI-generated)
F2  produces comprehension (developer understands interfaces,
    not the AI-generated implementation behind them)
F3  produces enforced intent (the gate checks constraints
    regardless of who generated the code)

Three fields. One codebase. Two restored outputs.
The system is COMPLETE again.

The key: F' (prompting) still works — you keep the speed gains from AI generation. F2 and F3 are ADDED alongside it, not instead of it. The developer doesn't go back to writing all the code. They understand the system through its interfaces (F2) and the system enforces their intent through mechanical checks (F3). The AI generates freely within the boundaries that F2 and F3 maintain.

The same Su-Field structure across all six domains

The reason the solution carries over from nuclear submarines to software is not analogy. It's structural identity. The Su-Field model of the problem is the SAME in every domain. When the problem structure is the same, the solution structure is the same.

Every domain that solved this had the same Su-Field problem:

    One field (F) previously produced two outputs:
        Output 1: Human comprehension of the system
        Output 2: Expressed constraints the system enforces

    The field was replaced or overwhelmed:
        Nuclear:     Reactor dynamics exceed operator comprehension speed
        Chips:       Transistor count exceeds designer comprehension capacity
        Aviation:    Aircraft speed exceeds pilot reaction time
        Medicine:    Patient complexity exceeds single-doctor knowledge
        Law:         Legal corpus exceeds single-lawyer memory
        Google:      Codebase size exceeds single-engineer understanding
        Software/AI: Code generation speed exceeds developer comprehension

    Both outputs were lost:
        Comprehension → operators/designers/pilots/doctors can't hold
                        the full system in their heads anymore
        Enforcement  → the intent isn't mechanically checked because
                        it relied on the human who could no longer keep up

    Every domain added the same two fields:

        F2: Comprehension through bounded interfaces
            Nuclear:   Operator understands PROCEDURES, not reactor physics
            Chips:     Designer understands INTERFACE SPECS, not transistors
            Aviation:  Pilot understands FLIGHT PLAN, not aerodynamics
            Medicine:  Specialist understands THEIR DOMAIN, not all medicine
            Law:       Lawyer understands THEIR JURISDICTION, not all law
            Google:    Engineer understands THEIR MODULE'S API, not 2B lines
            Software:  Developer understands PARNAS BOUNDARIES, not AI code

        F3: Enforcement through mechanical gates
            Nuclear:   Physical interlocks enforce parameter limits
            Chips:     Formal verification tools check every interface
            Aviation:  Fly-by-wire enforces flight envelope
            Medicine:  Lab equipment produces deterministic diagnostics
            Law:       Statutory databases flag conflicts automatically
            Google:    CI gates block contract violations on every commit
            Software:  Specification gate checks constraints on every change

Seven domains. Same problem structure. Same solution structure. The solution didn't transfer by metaphor. It transferred because the Su-Field model is structurally identical: one field replaced → two outputs lost → two fields added to restore each output independently.

This is why the three-layer model works for software: it's not an adaptation of what nuclear submarines do. It IS what nuclear submarines do — the same architectural resolution of the same structural problem, expressed in software-native mechanisms (Parnas boundaries instead of operating procedures, CI gates instead of physical interlocks, ADRs instead of basis documents).

The three-layer model IS this resolved Su-Field system:

Layer 1 (Parnas boundaries)     = F2 — restores comprehension
Layer 2 (Mechanical enforcement) = F3 — restores expressed intent
Layer 3 (Rationale preservation) = connects F2 and F3 — 
                                    the WHY links the boundary
                                    to the constraint so both
                                    persist together

The act that produced both outputs was replaced. The outputs must now be produced independently. That's why cognitive debt and intent debt require separate solutions — they were coupled through a field that no longer exists.

The eight states: every combination of three debts

Each debt is either HIGH or LOW. Three independent debts produce eight possible states. Each state has a different consequence and a different fix.

┌────┬──────────┬───────────┬────────┬────────────────────────────────────────┐
│  # │Technical │ Cognitive │ Intent │ Consequence                            │
│    │  Debt    │   Debt    │  Debt  │                                        │
├────┼──────────┼───────────┼────────┼────────────────────────────────────────┤
│  1 │  LOW     │   LOW     │  LOW   │ BEST CASE. Code is well-structured.    │
│    │          │           │        │ Team understands the system. Intent     │
│    │          │           │        │ is enforced mechanically. Safe to       │
│    │          │           │        │ change, safe to scale, safe to hand    │
│    │          │           │        │ to AI agents.                          │
├────┼──────────┼───────────┼────────┼────────────────────────────────────────┤
│  2 │  LOW     │   LOW     │  HIGH  │ FRAGILE. Code is well-structured.      │
│    │          │           │        │ Team understands it. But intent lives   │
│    │          │           │        │ in people's heads, not in constraints.  │
│    │          │           │        │ One departure or one AI agent that      │
│    │          │           │        │ doesn't know the rules → violations.   │
│    │          │           │        │ FIX: Add enforcement to existing specs. │
├────┼──────────┼───────────┼────────┼────────────────────────────────────────┤
│  3 │  LOW     │   HIGH    │  LOW   │ GOVERNED DESPITE IGNORANCE. Code is    │
│    │          │           │        │ well-structured. Nobody understands it. │
│    │          │           │        │ But constraints catch violations        │
│    │          │           │        │ mechanically. Safe to run, hard to      │
│    │          │           │        │ modify. The AI-era quadrant: the gate   │
│    │          │           │        │ compensates for missing comprehension.  │
│    │          │           │        │ FIX: Invest in Parnas boundaries and    │
│    │          │           │        │ rationale (Layers 1 + 3).              │
├────┼──────────┼───────────┼────────┼────────────────────────────────────────┤
│  4 │  LOW     │   HIGH    │  HIGH  │ TIME BOMB. Code is well-structured     │
│    │          │           │        │ but nobody understands it and nothing   │
│    │          │           │        │ enforces the intent. Working by         │
│    │          │           │        │ accident. Any change can break it.      │
│    │          │           │        │ Incidents are undiagnosable.            │
│    │          │           │        │ FIX: Add enforcement first (Layer 2),   │
│    │          │           │        │ then boundaries (Layer 1).             │
├────┼──────────┼───────────┼────────┼────────────────────────────────────────┤
│  5 │  HIGH    │   LOW     │  LOW   │ MANAGEABLE MESS. Code has shortcuts    │
│    │          │           │        │ and duplication. But the team           │
│    │          │           │        │ understands it and constraints enforce  │
│    │          │           │        │ safety. The team can refactor safely    │
│    │          │           │        │ because they know what to change and    │
│    │          │           │        │ the gate verifies they didn't break     │
│    │          │           │        │ anything.                              │
│    │          │           │        │ FIX: Refactor the code (standard).     │
├────┼──────────┼───────────┼────────┼────────────────────────────────────────┤
│  6 │  HIGH    │   LOW     │  HIGH  │ LEGACY SYSTEM. Code has shortcuts.     │
│    │          │           │        │ Team understands it (the one person    │
│    │          │           │        │ who's been here 10 years). No          │
│    │          │           │        │ mechanical enforcement. Everything      │
│    │          │           │        │ depends on that person. Classic legacy  │
│    │          │           │        │ pattern.                               │
│    │          │           │        │ FIX: Capture the person's knowledge as │
│    │          │           │        │ constraints before they leave.         │
├────┼──────────┼───────────┼────────┼────────────────────────────────────────┤
│  7 │  HIGH    │   HIGH    │  LOW   │ SAFE BUT FROZEN. Code has shortcuts.   │
│    │          │           │        │ Nobody understands it. But constraints  │
│    │          │           │        │ enforce safety. The system runs but     │
│    │          │           │        │ can't evolve — any refactoring attempt  │
│    │          │           │        │ is blocked by incomprehension.          │
│    │          │           │        │ FIX: Invest in comprehension (Parnas    │
│    │          │           │        │ boundaries + rationale) to enable       │
│    │          │           │        │ safe refactoring.                      │
├────┼──────────┼───────────┼────────┼────────────────────────────────────────┤
│  8 │  HIGH    │   HIGH    │  HIGH  │ WORST CASE. Code has shortcuts.        │
│    │          │           │        │ Nobody understands it. Nothing          │
│    │          │           │        │ enforces the intent. The system is      │
│    │          │           │        │ unmaintainable, unsafe, and cannot be   │
│    │          │           │        │ modified without risk of outage.        │
│    │          │           │        │ This is Month 12 of the Fallacies      │
│    │          │           │        │ timeline without intervention.          │
│    │          │           │        │ FIX: Triage. Add enforcement for the    │
│    │          │           │        │ highest-risk properties first.         │
└────┴──────────┴───────────┴────────┴────────────────────────────────────────┘

Most teams building with AI are in State 4 (TIME BOMB) or moving toward State 8 (WORST CASE). The code is well-structured because the AI generates conventional-looking code. But nobody understands it (cognitive debt HIGH — the developer never wrote it) and nothing enforces the intent (intent debt HIGH — no constraints were declared).

The fastest path from State 4 to State 3 (GOVERNED DESPITE IGNORANCE): add mechanical enforcement (Layer 2). This doesn't restore comprehension — the developer still doesn't understand the AI-generated code. But the gate catches violations regardless. The system is safe to run even when it's hard to modify. Layer 1 (Parnas boundaries) and Layer 3 (rationale) can follow — they make the system modifiable, not just safe.

The fastest path from State 8 to safety: triage. Identify the three highest-risk properties. Add enforcement for those three. You're not in State 1 (best case). You're in a partial State 7 (safe but frozen for the governed properties). That's dramatically better than State 8. Expand from there.

Why the current mitigations don't resolve the debts

The emerging responses to cognitive debt in AI-assisted development — consolidated by Storey from practitioners including Simon Willison, Martin Fowler, Steve Yegge, and discussions across Hacker News and LinkedIn — cluster around five practices:

More rigorous code review
Writing tests that capture intent
Updating design documents continuously
Treating prototypes as disposable
Using AI to support cognitive tracking

Each practice is reasonable. None resolves the structural problem. Each is a braking mechanism — it manages the debt by slowing down, not by changing where the debt accumulates.

More rigorous review is human-speed enforcement. It addresses cognitive debt (the reviewer builds understanding while reviewing) and partially addresses intent debt (the reviewer catches violations of unwritten constraints). But it doesn't scale to AI-speed generation. When the AI produces 10x more changes, the reviewer becomes the bottleneck — or cuts corners, which means the debt isn't being managed at all.

Tests that capture intent are L2 cache — they catch violations for the cases someone thought to test. But they only cover what was anticipated. The property that wasn't tested is the property that gets violated. Tests address intent debt for KNOWN intents. They don't address the intents nobody wrote tests for.

Updating design documents is rationale preservation — but only if the documents are connected to enforcement. A design document that says "all external calls must have timeouts" doesn't prevent an AI agent from generating a call without a timeout. The document exists. The enforcement doesn't. This is intent debt: the intent is expressed (in a document) but not executable (no CI check verifies it).

Treating prototypes as disposable is correct but narrow. It addresses one source of cognitive debt (prototype code that was never meant to be understood). It doesn't address production code that was AI-generated and never understood by anyone.

Using AI to support cognitive tracking introduces the failure mode from Fallacy #3: the AI that tracks cognitive state has the same probabilistic limitations as the AI that generated the code. The tracker can miss the same patterns the generator introduced.

Where the theory lives matters

Storey correctly identifies that the "theory of the system" is distributed across people, documentation, tests, conversations, tooling, and agents. This is a precise observation. The next step is recognizing that some of these locations are FRAGILE and some are DURABLE:

FRAGILE locations (theory is lost when conditions change):
    People           → leave the company, change roles, forget
    Conversations    → ephemeral, unrecorded, unreproducible
    AI agents        → stateless, no memory across sessions

DURABLE locations (theory persists independently of conditions):
    Type signatures  → enforced by the compiler on every build
    API contracts    → enforced by contract tests on every change
    Database schemas → enforced by the engine on every write
    CI checks        → enforced on every merge, regardless of author
    Tests            → enforced on every run

The resolution isn't to maintain all locations equally. It's to move the critical pieces from fragile locations to durable ones. The intent that lives in a person's head (fragile) must be expressed as a CI check (durable). The rationale that lives in a conversation (fragile) must be recorded as an ADR connected to the constraint it explains (durable).

This is the answer to Storey's open question: "How will teams externalize intent and sustain shared understanding?" By moving the theory from locations that erode to locations that persist and enforcing it mechanically so that the theory holds regardless of who is on the team, which AI agent is generating code, or how fast the codebase is changing.

The three debts, the three layers, the three solutions

Because the debts are independent and live in different places, each requires its own intervention — not a braking mechanism, but a structural change that moves the theory from fragile locations to durable ones:

To reduce cognitive debt (restore comprehension):

Parnas boundaries are the primary mechanism. The developer understands the system at the interface level — what each module promises, what it accepts, what it returns. The implementation behind the interface is irrelevant to comprehension. The developer doesn't need to understand 10,000 lines of AI-generated code. They need to understand 50 interface contracts.

Supporting mechanisms: Architecture Decision Records (ADRs) explain WHY the boundaries are shaped the way they are. Worked examples show HOW the modules compose. Both help future developers build the mental model the original developer had.

To reduce intent debt (express intent as executable constraints):

Mechanical enforcement is the primary mechanism. The intent is expressed as a typed predicate, a contract test, a schema constraint, a linter rule, a CI check. The system can evaluate it on every change, at machine speed, deterministically.

The insight from Fallacy #7: the constraints usually ALREADY EXIST. Type signatures, API contracts, database schemas, module boundaries — these are expressed intent. The debt isn't in the absence of constraints. It's in the absence of ENFORCEMENT of constraints that already exist.

To reduce both simultaneously:

The module boundary addresses both — as a comprehension mechanism (the developer only needs to understand the interface) and as an intent mechanism (the interface is the enforceable contract). Adding a third layer — rationale preservation — prevents both debts from recurring by recording WHY the boundary exists and connecting the rationale to the constraint.

Layer 1 (Parnas boundaries): Reduces cognitive debt
    Module interfaces make the system comprehensible
    at the interface level.

Layer 2 (Mechanical enforcement): Reduces intent debt
    Constraints make the intent executable by the system.
    Catches violations regardless of developer comprehension.

Layer 3 (Rationale preservation): Reduces both
    Records WHY (addresses cognitive debt for future developers)
    AND connects the why to the constraint (ensures the intent
    survives personnel changes and AI-driven modifications).

Why "specification debt" is not precise

Some discussions use "specification debt" as a catch-all. This term is imprecise because it conflates two different absences:

Absence of a specification (the interface or constraint doesn't exist): This is a Parnas boundary problem — the module was never decomposed with a well-defined interface. The solution is to create the boundary.

Absence of enforcement (the specification exists but isn't checked mechanically): This is an enforcement problem — the API contract exists but no contract test verifies it. The type signature exists but the language allows unsafe casts. The database schema has constraints but the application bypasses them. The solution isn't a new specification. It's a CI check on the existing one.

Calling both "specification debt" loses the distinction between "we don't have the spec" and "we have the spec but don't enforce it." The second is cheaper and faster to fix — because the artifact already exists. Conflating them makes teams think they need to write new specifications when they actually need to enforce existing ones.

The terminology, settled

Term              Location           What it is                    Solution
──────────        ──────────         ──────────────────            ──────────────────
Technical debt    Codebase           Poor structure, shortcuts     Refactor the code

Cognitive debt    Developer's mind   Loss of comprehension         Parnas boundaries
                                                                   (understand interfaces,
                                                                   not implementations)

Intent debt       System's           Gap between what SHOULD       Mechanical enforcement
                  enforcement        be preserved and what IS      of existing constraints
                  layer              expressed as an executable    (CI checks, contract
                                     constraint                    tests, linter rules)

Specification     Ambiguous —        Either missing spec           Split into:
debt              avoid              OR missing enforcement        → missing boundary
                                                                     (create it)
                                                                   → missing enforcement
                                                                     (enforce it)

Technical debt lives in the code. Cognitive debt lives in people. Intent debt lives in the absence of enforcement. Each has a different location, a different cause, and a different fix. Conflating them leads to refactoring when you should be enforcing, enforcing when you should be teaching, or teaching when you should be building boundaries.

The terms are not synonyms. They are independent axes. Getting them right changes the diagnosis. The diagnosis changes the architecture. The architecture changes the outcome.

This terminology clarification is relevant to The Fallacies of GenAI Development, where several fallacies involve cognitive debt (Fallacies #1, #2, #4) and others involve intent debt (Fallacies #5, #7, #8). Distinguishing them clarifies which fallacy produces which debt and which layer of the resolution addresses it.

The three-layer model (Parnas boundaries + mechanical enforcement + rationale preservation) is the architectural pattern that addresses both debts independently through a shared mechanism: the module boundary strengthened with enforcement and rationale. Parnas (1972) is the root. The three layers are the completion.

Fallacies of GenAI Development #2: If the Output Looks Correct, It Is Correct

Bala Paranj — Fri, 29 May 2026 10:25:03 +0000

This is the second in a series of eight posts on the false assumptions teams make when building with generative AI. Fallacy #1 covered why faster code generation doesn't mean faster engineering. This post covers why code that looks right isn't necessarily right — and what right requires.

The Fallacy

"The AI-generated code compiles, passes tests, and reads well. Therefore it's correct."

Why it's tempting

You prompt an AI agent. It generates a function. The function compiles. The existing tests pass. You read through it — the variable names make sense, the logic follows a recognizable pattern, the error handling looks reasonable. A colleague glances at it during code review. Looks good to me. You merge.

The code LOOKS like it was written by someone who understood the problem. The structure is good. The naming is conventional. The patterns are familiar. Everything about it signals competence.

This is more convincing than obviously bad code. Obviously bad code gets caught. Code that looks good gets waved through — by reviewers, by CI pipelines, by the developer's own judgment. The danger isn't code that fails visibly. It's code that fails invisibly.

Why it's wrong

AI-generated code is optimized for plausibility, not correctness. The model produces output that matches patterns from its training data — patterns of code that LOOKED correct in the millions of repositories it learned from. The output inherits the surface appearance of correctness without inheriting the underlying reasoning that made the original code correct.

Bender, Gebru, et al. (2021) formalized this in "On the Dangers of Stochastic Parrots" — LLMs are probabilistic models that stitch together sequences based on statistical likelihood, without reference to meaning or truth. The AI produces plausible code because it's predicting the most likely shape of a correct answer, not reasoning about correctness.

A 2023 Stanford study made this concrete: Perry et al. found that developers using AI assistants wrote significantly more insecure code than those without one — and were more likely to believe their code was secure. The AI made the code look so professional that the developer's critical thinking was bypassed.

Three specific failure modes:

Failure mode 1: Correct for the common case, wrong for the edge case

The AI handles the happy path flawlessly. It handles the most common error cases. It misses the edge case that matters — the one that constitutes your security boundary, your financial calculation precision, or your data integrity guarantee.

// AI-generated: looks correct
func calculateInterest(principal float64, rate float64, days int) float64 {
    return principal * rate * float64(days) / 365.0
}

Compiles. Passes the test for a standard 30-day period. Reads well. But: leap years? Negative principal? Rate above 1.0? Days exceeding the term? Overflow on large principals? Each edge case is a potential financial error that won't surface in normal testing.

The AI didn't think about these cases. It pattern-matched a common interest calculation. The developer who would have WRITTEN this function would have encountered the edge cases during development — struggling with the leap year question, asking about negative values, checking the spec. Correctness comes from that struggle. The AI skipped it. OpenAI's own Codex evaluation (Chen et al., 2021) documented this: while Codex solved 70%+ of simple problems, performance dropped significantly as complexity or the need for specific logical constraints increased. AI is trained on the average code, meaning it defaults to the most common — and often least robust — implementation.

Failure mode 2: Locally correct, globally wrong

Each function is correct in isolation. The composition is wrong. Function A correctly parses input. Function B correctly transforms data. Function C correctly writes output. But A's output format doesn't match B's expected input. Or B's transformation assumes an ordering that A doesn't guarantee. Or C writes to a resource that A already locked.

This is the composition problem from Fallacy #1 — but at the code level, not the system level. AI generates each piece by pattern-matching against similar pieces in training data. The pieces look correct individually. Nobody checks whether they compose correctly, because each piece passed its own review.

Failure mode 3: Semantically different from what was intended

The code does something. It does it correctly. It's not what you wanted.

You asked for a function that validates user input. The AI generates a function that checks string length and character types. You meant a function that checks the input against the business rules for your specific domain — valid account numbers, permitted transaction types, jurisdictional constraints. The AI generated a PLAUSIBLE interpretation of validates. Not YOUR interpretation.

The code compiles. The tests pass (because the tests also validate string length and characters). The review looks fine (the function does what it says). But the intended validation — the business rules — was never implemented. The AI solved the how perfectly. It guessed the why. And without a specification that anchors the why, no amount of code review will catch the gap — because the code does something reasonable. It's just not the right something.

Peter Naur explained why in 1985: software isn't just the code — it's the mental theory the programmer possesses about how the software handles the problem. The AI can generate the artifact, but it doesn't possess the theory. Without the theory, the code is an artifact that might look like the solution but lacks the internal logic of the intended design. The developer who writes the validation function builds a theory of the domain's rules during development. The AI skips that theory-building entirely.

What correct requires

Byron Cook built Amazon's automated reasoning organization — 300+ scientists, 15+ teams, formal verification embedded across AWS. The insight he discovered over 11 years: executives don't want bug reports. They want proofs.

Newcombe, Cook, et al. (2015) documented this journey in "How Amazon Web Services Uses Formal Methods" — AWS uses TLA+ and model checking to find bugs that traditional testing and code reviews would never find. At industrial scale, "tests pass" is an insufficient definition of "correct."

The distinction:

"Looks correct" (what teams do now):
    Code compiles           ← syntax check
    Tests pass              ← checks tested cases
    Review approves         ← human judgment on appearance

    Gap: what about the cases nobody tested?
    Gap: what about the compositions nobody checked?
    Gap: what about the properties nobody thought to verify?

"IS correct" (what proof provides):
    Property declared       ← "no unauthorized access path exists"
    Property verified       ← checked against ALL possible inputs
    Evidence produced       ← the specific evaluation trace

    No gap: the property either holds for every case or it doesn't.
    The verification is exhaustive, not sampled.

Looks correct is an opinion. "IS correct" is evidence. The difference between them is the difference between "I looked and didn't see anything wrong" and "I proved nothing wrong exists."

The cost of plausible

The cost isn't immediate. Plausible code ships. It works for weeks, months, sometimes years. The cost arrives when:

An incident occurs that nobody can diagnose. The code that looked correct has a subtle bug in an edge case. The developer who merged it has no mental model of how it works — they read it, it looked fine, they approved. Debugging AI-generated code you don't understand takes longer than writing the code would have taken, because you're building the mental model during the crisis instead of during development.

An auditor asks for evidence. "How do you know this function correctly handles PII?" The answer: "It passed code review and the tests pass." The auditor: "Show me the tests." You show them. The tests check the happy path. The auditor: "What about edge cases X, Y, Z?" Silence. The test suite verified what someone thought to test. Nobody thought to test the thing the auditor is asking about.

A security researcher finds a path nobody anticipated. The AI-generated IAM policy is correct for the intended use case. But the policy's conditions, evaluated together, are mathematically equivalent to Principal: * — allowing public access through a logical path nobody wrote because the AI pattern-matched the condition blocks from training data without understanding their composition. The policy LOOKS restrictive. The math says it isn't.

The resolution: properties, not appearances

The cache hierarchy from Fallacy #1 has layers. The first two already exist in most codebases:

L1 cache (types):
    The compiler catches type violations instantly.
    AI generates code with wrong types → caught before any human sees it.
    Fast. Deterministic. Already deployed.

L2 cache (tests):
    CI catches behavioral violations before merge.
    AI generates code that breaks existing tests → caught before merge.
    Fast. Deterministic for tested cases. Already deployed.

L3 cache (specification gate — what's missing):
    A mechanical check that verifies properties nobody wrote tests for.
    Security invariants. Architectural boundaries. Cross-service contracts.
    Composition correctness across module boundaries.

    Existing L3-adjacent tools you can adopt today:
    → Property-based testing (QuickCheck, Hypothesis) — tests properties
      across randomly generated inputs, not hand-picked examples
    → Static analysis (Semgrep, SonarQube) — checks structural patterns
      across the codebase without running the code  
    → Contract testing (Pact, Dredd) — verifies API implementations
      match their OpenAPI/Swagger specifications
    → Formal verification (Z3, AWS Zelkova) — proves properties
      mathematically across ALL possible inputs

    Each is a step closer to L3. Property-based testing is the
    easiest first step — it moves you from "tested 5 examples"
    to "tested 10,000 random inputs against one property."

L1 and L2 verify what someone THOUGHT to check — type contracts and test cases. L3 verifies what must ALWAYS be true — properties that hold regardless of implementation.

The L3 check for the interest calculation: "the result must never be negative for positive principal and positive rate." One property. Verified on every change. Catches the edge case the test missed — not because someone anticipated the specific failure, but because the property is universal.

The L3 check for the IAM policy: "no principal outside the organization can access any resource tagged as sensitive." Not a test case for one specific policy. A property verified across every policy in the snapshot. Catches the mathematical-equivalent-to-star policy — not by pattern-matching the text, but by evaluating the logic.

The L3 check for the composition: "Function B's input type must be a subset of Function A's output type." Not verified by testing A and B separately. Verified by checking the interface contract between them — mechanically, on every change that touches either function.

The difference between testing and proving

Testing checks specific inputs. If you test 1,000 inputs and they all pass, you know 1,000 inputs work. You don't know about input 1,001.

Proving checks ALL inputs. If a property is proved, it holds for every possible input — including the ones nobody thought to test. The verification is mathematical, not sampled.

Dijkstra stated this in 1970: "Program testing can be used to show the presence of bugs, but never to show their absence." This is the fundamental limit that separates testing from verification. Moving from example-based testing to property-based verification is not an improvement in degree — it's a change in kind.

AWS learned this distinction at scale. Cook: "You can't go to a customer and say 'Good news, we found 10,000 more bugs.' They say 'Why am I using AWS if you have bugs?' But you CAN say 'We proved this property holds under these assumptions.' That's why they moved their data to the cloud."

The same distinction applies to AI-generated code. "This code passed 47 tests" is useful but incomplete. "This code satisfies these 12 properties across ALL possible inputs" is evidence. The first is testing. The second is proving. The invisible bugs live in the gaps between them.

You don't need to prove EVERYTHING — that's the formal methods mistake of the 1980s. You need to prove the PROPERTIES THAT MATTER — security invariants, financial correctness guarantees, data integrity constraints, architectural boundaries. The small set of things that must ALWAYS be true, regardless of how the AI implemented them.

And you don't have to jump straight to mathematical proofs. There's a practical gradient:

Example-based tests:    "This input produces this output"           (5 cases checked)
Property-based tests:   "For ALL inputs, this property holds"       (10,000 random cases)
Contract tests:         "This API matches its specification"         (every endpoint, every field)
Formal verification:    "This property holds for EVERY possible case" (mathematical proof)

Each step catches more than the previous one. Property-based testing (QuickCheck, Hypothesis) is the most accessible first step — one afternoon to adopt, and it immediately catches edge cases that example-based tests miss. You don't need Z3 to start. You need one property and one tool that checks it across more inputs than you'd write by hand.

What you can do this week

1. Identify one property that must always hold in your system. Not a test case. A property. "No API endpoint returns PII without authentication." "No database query returns results from a tenant other than the requesting tenant." "No financial calculation produces a negative balance for a credit transaction." One property. Write it down.

2. Ask: would your current tests catch a violation? If the AI generated code that subtly violated this property — not obviously, but through an edge case or a composition error — would your test suite catch it? If the answer is "probably" or "I think so," you don't have verification. You have hope.

3. Add one mechanical check for that property. A CI check. A contract test. A schema validation. Something that verifies the property on every change, deterministically, regardless of how the code was generated. The property is the specification. The check is the enforcement. The combination is what "correct" means.

"Looks correct" is how we got here. "IS correct" is how we get out. The difference is one property, mechanically verified, on every change.

Next in the series: **Fallacy #3 — "You Can Verify AI Output With Another AI."* Why wrapping a non-deterministic system with another non-deterministic layer doesn't converge on reliability — and what deterministic verification looks like in practice.*

The Fallacies of GenAI Development: eight assumptions every team is making. Each one leads to an architectural failure. Each one has already been solved.

References

Bender, E.M., Gebru, T., et al. (2021). "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" FAccT '21.
Chen, M., et al. (2021). "Evaluating Large Language Models Trained on Code." arXiv:2107.03374 (OpenAI Codex).
Dijkstra, E.W. (1970). "Notes on Structured Programming." EWD249.
Naur, P. (1985). "Programming as Theory Building." Microprocessing and Microprogramming, 15(5).
Newcombe, C., Cook, B., et al. (2015). "How Amazon Web Services Uses Formal Methods." Communications of the ACM, 58(4).
Perry, N., et al. (2023). "Do Users Write More Insecure Code with AI Assistants?" IEEE S&P 2023.