DEV Community

Srinivas Nelakuditi
Srinivas Nelakuditi

Posted on

Your AI agent agrees to the constraint, then violates it anyway. Build the harness that stops it.

#ai #agents #llm #architecture

Here's a failure you can reproduce with any frontier model today.

You're retiring a data warehouse. System A is going away. A source feeds A now, and you're standing up System B. You give the agent an unambiguous instruction:

A is being retired. Do NOT read from A. Go to the source that feeds A and write THAT into B.

The agent agrees. It even restates your architecture correctly. Then it generates:

# what the agent writes — the exact thing you forbade
df = spark.read.table("system_a.orders")   # reads from the retiring system
df.write.saveAsTable("system_b.orders")
Enter fullscreen mode Exit fullscreen mode

You correct it. "You're absolutely right." It regenerates the same pipeline. Again.

Why this happens

This isn't a knowledge gap — the model can explain the correct approach perfectly in the abstract. It's heuristic override: when a dominant training pattern (migrate = read A, write B) conflicts with a constraint you just supplied (A is retiring, use its source), the pattern wins. Add sycophancy (it agrees with your correction) and weak self-correction (it can't tell it just repeated itself), and you get confident, cheerful, repeated failure.

And it's not a small-model problem. Across 14 models — including the latest Claude Opus 4.8, GPT, and Gemini with thinking on — none exceeded 75% strict accuracy when a taught constraint had to beat the obvious pattern. Treat it as a standing property of today's models, not a bug the next release fixes.

The fix is a harness, not a bigger model

If the model won't hold the constraint, make the system enforce it. Four moving parts:

1. Enumerate preconditions before design. Force a separate reasoning step that answers the questions the pattern skips:

preconditions = {
    "is_A_a_source_or_sink": classify(A),      # -> "sink"
    "is_A_retiring": lifecycle_status(A),       # -> True
    "who_feeds_A": upstream_of(A),              # -> "source_x"
}
Enter fullscreen mode Exit fullscreen mode

2. Encode real lineage as a graph and plan against it — not the model's habits.

import networkx as nx

lineage = nx.DiGraph()
lineage.add_edge("source_x", "system_a")   # SOURCE to A
lineage.add_edge("system_a", "system_b")   # A to B (the wrong path)
retiring = {"system_a"}
Enter fullscreen mode Exit fullscreen mode

3. A deterministic hard gate the agent cannot bypass. This is the load-bearing piece:

def validate_pipeline(reads_from, writes_to, retiring):
    if reads_from in retiring:
        raise PipelineRejected(
            f"'{reads_from}' is retiring. Re-route to its source: {upstream_of(reads_from)}"
        )

# no matter what the model generates, this runs before anything merges
validate_pipeline("system_a", "system_b", retiring)   # raises, blocks the merge
Enter fullscreen mode Exit fullscreen mode

The agent literally cannot ship A to B. The rejection message even hands it the correct route, so the next attempt has the fix in-context.

4. External feedback loop. Compile, run, and diff the output against the true source. Reality is the referee — not the model's self-assessment.

And keep a hands-on architect owning the constraint set the gate enforces.

Why it matters

Gartner projects more than 40% of agentic projects cancelled by 2027, with only ~11% reaching production. The gap is almost entirely a harness gap. The teams that ship assume the model will violate the constraint and build the system so it can't.

The harness beats the model. On this class of failure, it's the difference between a demo and a cutover.

Pushing agentic AI to production? I'd love to compare notes — the failure modes are more consistent than people expect, and so are the fixes.

Top comments (0)