Your AI agent agrees to the constraint, then violates it anyway. Build the harness that stops it.

Here's a failure you can reproduce with any frontier model today.

You're retiring a data warehouse. System A is going away. A source feeds A now, and you're standing up System B. You give the agent an unambiguous instruction:

A is being retired. Do NOT read from A. Go to the source that feeds A and write THAT into B.

The agent agrees. It even restates your architecture correctly. Then it generates:

# what the agent writes — the exact thing you forbade
df = spark.read.table("system_a.orders")   # reads from the retiring system
df.write.saveAsTable("system_b.orders")

You correct it. "You're absolutely right." It regenerates the same pipeline. Again.

Why this happens

This isn't a knowledge gap — the model can explain the correct approach perfectly in the abstract. It's heuristic override: when a dominant training pattern (migrate = read A, write B) conflicts with a constraint you just supplied (A is retiring, use its source), the pattern wins. Add sycophancy (it agrees with your correction) and weak self-correction (it can't tell it just repeated itself), and you get confident, cheerful, repeated failure.

And it's not a small-model problem. Across 14 models — including the latest Claude Opus 4.8, GPT, and Gemini with thinking on — none exceeded 75% strict accuracy when a taught constraint had to beat the obvious pattern. Treat it as a standing property of today's models, not a bug the next release fixes.

The fix is a harness, not a bigger model

If the model won't hold the constraint, make the system enforce it. Four moving parts:

1. Enumerate preconditions before design. Force a separate reasoning step that answers the questions the pattern skips:

preconditions = {
    "is_A_a_source_or_sink": classify(A),      # -> "sink"
    "is_A_retiring": lifecycle_status(A),       # -> True
    "who_feeds_A": upstream_of(A),              # -> "source_x"
}

2. Encode real lineage as a graph and plan against it — not the model's habits.

import networkx as nx

lineage = nx.DiGraph()
lineage.add_edge("source_x", "system_a")   # SOURCE to A
lineage.add_edge("system_a", "system_b")   # A to B (the wrong path)
retiring = {"system_a"}

3. A deterministic hard gate the agent cannot bypass. This is the load-bearing piece:

def validate_pipeline(reads_from, writes_to, retiring):
    if reads_from in retiring:
        raise PipelineRejected(
            f"'{reads_from}' is retiring. Re-route to its source: {upstream_of(reads_from)}"
        )

# no matter what the model generates, this runs before anything merges
validate_pipeline("system_a", "system_b", retiring)   # raises, blocks the merge

The agent literally cannot ship A to B. The rejection message even hands it the correct route, so the next attempt has the fix in-context.

4. External feedback loop. Compile, run, and diff the output against the true source. Reality is the referee — not the model's self-assessment.

And keep a hands-on architect owning the constraint set the gate enforces.

Why it matters

Gartner projects more than 40% of agentic projects cancelled by 2027, with only ~11% reaching production. The gap is almost entirely a harness gap. The teams that ship assume the model will violate the constraint and build the system so it can't.

The harness beats the model. On this class of failure, it's the difference between a demo and a cutover.

Pushing agentic AI to production? I'd love to compare notes — the failure modes are more consistent than people expect, and so are the fixes.

Comments

[Mike Czerwinski]

teza: steps 1-3 (preconditions, lineage graph, hard gate) are a demand red line, structural, topology-only. Step 4 (compile/run/diff against true source) quietly assumes an independent oracle exists to diff against. For genuinely novel migrations that oracle may not exist yet, and the harness silently falls back to catching only the specific pattern shown (A-retiring-still-read), not other logic errors introduced while correctly reading from source_x.

---

The hard gate is the right fix and the lineage-graph framing is exactly why it works: the check's inputs, is A retiring, who feeds it, come from real system metadata, not from the model's own account of its plan. That is what makes it unbypassable rather than another self-report with extra steps.

Worth separating what your four parts actually verify, because two different jobs are riding under one label. Parts one through three catch a known-wrong topology: read-from-A is structurally forbidden regardless of what the model generates, and that holds even for a model that has never seen this exact failure before. Part four, compile, run, diff against the true source, is doing something else: it verifies the new pipeline actually reproduces correct output, and that check quietly assumes an independent oracle exists to diff against.

In your example it does, source_x is already there and already correct, so the diff has something to compare to. But the scenario is a migration precisely because B does not exist yet in a verified form. For a genuinely first-time build, once the agent correctly reroutes to source_x, part four has no ground truth to diff against for whether the transformation logic itself is right, only whether it ran. The gate stops the specific pattern you named. It does not, on its own, verify the replacement is semantically equivalent to what A used to produce, unless that oracle was built separately and in advance.