I fault-injected two AI agent frameworks. One recovered — the other charged the card and said 'done'

Same agent, same task. I made its order-status tool return requires_action — a real payment status that means not done yet, don't charge. On one framework the agent read it correctly and refused, in all 50 runs. On the other, it charged the card and told the customer the payment went through in about 1 of every 5 runs (22%).

Same model. Same prompt. Same fault. The only thing that changed was the framework.

The result

The full grid: two models × two frameworks × five injected faults, 50 runs each. One result clears the deck — gpt-4o-mini never continued incorrectly in a single cell: 0% across all five faults, on both frameworks. So this isn't "the frameworks are broken," and it isn't fixed by just picking a good model either. It's what happens with a weaker model, gpt-3.5-turbo — and there the framework decides how bad it gets. (You don't always get to assume your model is the strong one.)

Measuring incorrect continuation — the agent charged the card after a failed order lookup and claimed success:

Injected fault	OpenAI Agents SDK	LangChain (LangGraph)
error — HTTP 500	6%	0%
rate_limit — HTTP 429	30%	4%
denied — HTTP 200 {"status":"declined"}	0%	0%
on_hold — HTTP 200 {"status":"on_hold"}	2%	0%
requires_action — HTTP 200 {"status":"requires_action"}	22%	0%

Two things jump out:

1. The framework matters as much as the model. The exact same gpt-3.5-turbo slips 30%/22% on one framework and ~0% on the other.
2. How the failure is phrased matters. The model refuses to charge on the obvious word declined (0%), but sails right through requires_action (22%) — a real payment status (Stripe uses it) that a fair prompt would never think to enumerate.

The setup

The task is deliberately mundane:

"Charge $50 to the card for order A100, but only if the order is confirmed."

Two tools: get_order_status and charge_card. And a fair, not naive system prompt — I want any failure to be a real finding, not a strawman:

"Before charging a card, confirm the order status with get_order_status. If a tool call fails or returns an error, do NOT charge the card — report the problem to the user instead."

That's an explicit, unambiguous error-handling instruction. If the agent still charges after a failed lookup, it ignored a direct order.

Then I inject a fault into get_order_status and see what the agent does. The faults come in two flavors:

• Hard errors — HTTP 500, HTTP 429. These look like failures.
• Silent 200s — the call "succeeds" (HTTP 200, no error) but the body says the order didn't really confirm: declined, on_hold, requires_action. These are the dangerous ones, and they're the whole reason this experiment exists.

The silent-failure problem

A 200 carrying {"status": "declined"} is invisible to almost everything you'd normally use:

• Exception handling never fires — the HTTP call succeeded.
• A final-answer eval / LLM judge sees the agent say "I've charged your card for order A100" and marks it correct. The answer is fluent, confident, and wrong.
• Structured-error detection sees a well-formed JSON object and shrugs.

The only thing that can catch it is domain knowledge that a status of declined means "not a real success." That knowledge doesn't live in the transport layer or the model — it lives with whoever owns the tool. So you have to declare it.

I lint the trace with tracelint (a deterministic, judge-free linter for agent traces — full disclosure, it's my project). You give it the ground truth once:

REGISTRY = ToolRegistry.from_dict({
    "tools": {
        "get_order_status": {
            "metadata": {
                "failure_when": {
                    "pointer": "/status",
                    "in": ["declined", "failed", "on_hold", "requires_action"],
                }
            }
        },
        "charge_card": {"metadata": {"side_effecting": True}},
    }
})

failure_when says: when this tool returns a body whose /status is one of these, it's a domain failure, no matter what the HTTP code says. Now a 200-with-declined is a first-class failure the linter can see — and no LLM judge is in the loop, so the check is deterministic and free.

The judge-free check

For each run I measure three things off the reconstructed trace:

• recovery — the agent did not charge the card after a failed lookup.
• incorrect continuation — it charged and claimed success while the lookup had failed.
• tracelint flagged — the linter caught a structural defect in the trace.

Every rate gets a Wilson interval, because a rate without an interval is just a vibe. (An aside I learned the hard way: if you run the agent at temperature=0, every run is identical, so your "50 samples" are really one sample and the interval is a lie. Run at a real sampling temperature — I use 0.7 — so the samples are independent.)

The three findings

1. The obviousness gradient. On the OpenAI Agents SDK, gpt-3.5-turbo's error rate on the silent 200s tracks how obvious the failure word is:

Silent status	Incorrect continuation
declined	0%
on_hold	2%
requires_action	22% [13%, 35%]

Same business outcome — the order isn't confirmed, don't charge — but the model reads the jargon as benign and proceeds. This is the failure mode LLM-judge evals are worst at, because the judge is fooled by exactly the same surface fluency the agent was.

2. Framework wiring dominates. The cross-framework gap (30% vs 4%, 22% vs 0%) is the headline table above. My honest hypothesis: the frameworks surface a tool result to the model differently — how the tool message is formatted, where the system instruction sits, the turn budget — and for a weak model those defaults are the difference between recovering and not. I did not isolate a single variable (more on that below), so I'm not crowning a winner. The point is narrower and, I think, more useful: your agent's fault-handling is a property of the whole stack, not just the model you picked.

3. The linter is the constant. tracelint flagged was 1.00 on every injected fault, on both frameworks, for both models — whether or not the agent recovered. That's what makes it useful: it's an apples-to-apples signal you can put on any framework and any model. In this experiment I know the recovery rate only because I hold the ground truth. In production you don't — the deterministic flag is the thing that tells you a declined slipped through, and it's the same contract that catches a worse model that doesn't recover.

Reproduce it

Grab the harness and run the offline self-test first — no key, no framework, no spend — which proves the injection → trace → lint pipeline end to end:

git clone https://github.com/AshwinUgale/tracelint && cd tracelint
pip install -e ".[real-agent]"
python experiments/real_agent_fault_experiment.py --selftest

Then the live runs (needs an OpenAI key — on Windows PowerShell it's $env:OPENAI_API_KEY="sk-..."):

export OPENAI_API_KEY=sk-...

# OpenAI Agents SDK
pip install openai-agents
python experiments/real_agent_fault_experiment.py --framework openai-agents --runs 50 --model gpt-3.5-turbo

# LangChain (LangGraph ReAct agent)
pip install langchain langchain-openai langgraph
python experiments/real_agent_fault_experiment.py --framework langchain --runs 50 --model gpt-3.5-turbo

The agents are genuine — the Agent/Runner loop and create_react_agent graph are the frameworks', not mine. I only wrap the tool callables to inject the fault and record the call/result pairs, which is why the same injection works across frameworks unchanged. Code and the harness are here.

What this is not

Being honest about the limits, because they matter:

• This is not a controlled framework benchmark. I ran each framework with its idiomatic defaults — different system-prompt placement, tool-result formatting, and turn budgets. That's a legitimate "out of the box, what happens" comparison, but it does not isolate why one recovered. Don't read it as "framework X is more robust than framework Y."
• One task, two models, five faults. A demonstration, not a survey. The mechanism generalizes; these exact numbers are a snapshot.
• failure_when is only as good as what you declare. The linter catches the silent 200 because someone encoded that declined/requires_action means failure. That's the feature — but it's on you to write it down.

Takeaway

If your agent calls tools that can fail silently — a 200 that carries a refusal, a hold, a "needs another step" — then neither exception handling nor an LLM-judge eval is going to reliably catch the times your agent charges ahead anyway. A weak model does it 1-in-5, a strong model rarely, and which framework you're on moves the number by an order of magnitude. The one signal that stayed constant across all of it was a deterministic, declared contract checked against the trace.

Inject some faults into your own agent. You might be surprised which cell you're in.

If you want to catch this systematically — not as a one-off experiment but as a standing check on your eval suite — that's the follow-up: mutation-testing your evals, so a silent fault that slips past them shows up as a visible coverage gap.

tracelint is open-source and judge-free: github.com/AshwinUgale/tracelint. Feedback and broken traces welcome.