AI Coding Failures Are Hardest to Catch

Imagine an AI-generated integration client that automatically retries requests after a timeout. The implementation is clean. The types are correct. The tests confirm that retries work. But the operation is not idempotent.

If the remote service completed the first request before the connection failed, the retry can perform the operation twice. In a payment workflow, that might mean a duplicate charge. The code did exactly what the prompt requested. The failure came from a constraint the prompt did not contain.

This is the difficult class of AI coding failures: code that is structurally valid and locally reasonable but wrong at the system level.

Three Levels of Correctness

It helps to distinguish between three forms of correctness.

1. Syntactic correctness: The code parses, compiles, and satisfies structural rules.
2. Local correctness: It performs the requested task under expected conditions.
3. System correctness: It preserves business rules, security boundaries, data invariants, architectural constraints, and operational behavior. Coding assistants can often achieve the first two. Most hidden risks appear in the third.

A model may know the framework, understand the local function, and follow nearby conventions. It still may not know that:

• Requests must be safe to retry.
• Records belong to separate tenants.
• NULL and false have different business meanings.
• A fallback response must remain distinguishable from an empty result.
• A particular abstraction exists because a previous implementation caused an incident.

The central problem is not that generated code always contains obvious errors. It is that plausible code can conceal assumptions.

When Error Handling Changes the Meaning of Failure

Consider this implementation:

try:
    return fetch_records()
except Exception:
    logger.warning("Fetch failed")
    return []

The function returns the expected type and prevents an exception from reaching the caller. It might even satisfy a test that expects an empty list. But it collapses two states:

• No records exist.
• Records could not be retrieved.

Downstream code can no longer distinguish a legitimate empty result from an operational failure. A dashboard may display zero activity, a reconciliation process may skip work, or another service may treat the request as successful.

The code did not crash. It changed the meaning of the system’s response. A stronger review question is therefore:

Which states does this implementation collapse, conceal, or reinterpret?

Common Quiet Failure Modes

Authorization implemented as filtering

Suppose an endpoint filters records by user_id but does not enforce tenant membership. Tests built around one tenant will pass. The defect appears only when an authenticated user supplies an identifier associated with another tenant.

This requires an authorization matrix, not just one authenticated and one unauthenticated test:

User state	Resource state	Expected result
Correct user and tenant	Owned resource	Allow
Correct user, different tenant	Existing resource	Deny
Delegated user with permission	Shared resource	Allow
Valid user without required role	Restricted resource	Deny

The matrix makes the actual security model visible.

Migrations that preserve shape but change meaning

A generated migration might replace every nullable Boolean with false. The migration executes successfully, but “unknown” has now become “no.”

Before approving the change, inspect existing values, document the state mapping, run the migration against production-shaped data, and compare invariants before and after execution. A successful migration command proves that the statements ran. It does not prove that the data retained its meaning.

Tests that repeat the implementation’s assumption

If an assistant generates code and tests from the same prompt, both may contain the same misunderstanding.

For example, the implementation might apply a discount before tax and generate tests that confirm that order. The suite passes because the tests agree with the code—not because either agrees with the billing policy.

For consequential behavior, expected results should come from an independent source: approved acceptance examples, a domain-owner decision table, or an existing contract.

Why Normal Controls Are Not Conclusive

Compilers, type systems, static analysis, tests, review, and staging remain essential. Each one answers a limited question.

A compiler establishes structural validity. Static analysis detects known patterns. A test confirms that an assertion held under a specific setup. Coverage shows which lines executed, not whether the correct behavior was asserted.

Code review has limits as well. If AI assistance increases implementation throughput, reviewers may receive more code than they can reconstruct carefully. The bottleneck moves from producing code to establishing confidence in it.

That does not make AI-generated code inherently worse than human-written code. Humans create the same categories of defect. AI changes how quickly plausible, unverified implementations can reach review.

Match Verification to the Risk

A formatting helper and an authorization change should not pass through identical controls.

Assess a material change using five factors:

• Criticality: What could be harmed?
• Uncertainty: Which requirements or assumptions remain unclear?
• Blast radius: How many users, records, or systems could be affected?
• Detectability: How quickly would incorrect behavior become visible?
• Reversibility: Can the change be disabled or repaired safely?

Use those answers to select a verification tier.

Tier 1: Low-risk and reversible

For isolated helpers, presentational changes, and developer tooling:

• Normal automated tests
• Standard peer review
• Confirmation that the change is isolated

Tier 2: Behaviorally significant

For business rules, caching, background jobs, and integrations:

• Acceptance examples derived from requirements
• Negative and boundary tests
• Verification of external contracts
• Domain-aware review
• Telemetry for important failure paths

Tier 3: High-impact or difficult to reverse

For authorization, payments, migrations, infrastructure, and concurrency-sensitive workflows:

• Security or domain-owner review
• Authorization or state-transition matrices
• Failure-injection or concurrency tests
• Rehearsal and reconciliation
• Controlled rollout
• A demonstrated rollback or repair plan

The tier should depend on potential consequences, not merely on whether a person or model produced the code.

Add an Evidence Package to the Pull Request

For a consequential AI-assisted change, require the pull request to answer:

1. What behavior is required?
2. What assumptions were made?
3. Which invariants could be violated?
4. What evidence supports correctness?
5. How will failure be detected and contained?

This does not require a separate governance process. A short, structured section in the pull request is enough.

The purpose is to move the review conversation away from “Does this look reasonable?” and toward “What evidence justifies releasing it?”

Code is not complete because it looks finished. It is complete when the team can explain what it must do, show why it is likely to do it, detect when it does not, and recover safely.

Which system invariant is most likely to be absent from the prompts your team currently gives coding assistants?