DEV Community: samarth

Part 2: We Fixed the Root Causes of Our AI Agent Incident. Routing Models by Step Cut Latency 86%

samarth — Tue, 20 Oct 2026 06:00:38 +0000

Note: this is a composite scenario built from common production failure patterns in agent systems. Ticket IDs, dollar amounts, latencies, and probabilities are illustrative. I drafted it with AI assistance and edited it myself.

In part 1, our billing agent settled a $48,000 credit it should have routed to a human. We showed why re-running the agent five times proved nothing, then used cut-point replay and a deterministic gate to prove the fix.

The gate stopped the bleeding. It did not explain why the agent reached the wrong conclusion in the first place. This post is about the upstream causes, and about the second half of the TICKET-8841 trace: why a six-step ticket took 42.6 seconds and cost $0.184.

Act III: Fixing the upstream root cause

The gate sits at Boundary 5. The agent got there because of what happened at Boundaries 2 and 3. When we held the post-mortem, someone immediately asked:

"Why are we still maintaining a RAG pipeline? Frontier models have 1M-token context windows now. Let's dump the entire 350-page Finance & Contract Handbook into the prompt."

It's a fair question, so we tested it.

Why "just stuff the handbook in" didn't win

Cost. A 350-page handbook is roughly 175,000 tokens, not 1M. Using an illustrative $3.00 per 1M input tokens:

175,000 tokens × $3.00 / 1M       = $0.525 per call
× 45,000 tickets/month            ≈ $23,600/month (uncached, one call per ticket)
with ~90% prompt-cache discount   ≈ $2,400/month

Our retrieval pipeline cost about $600/month. With caching, the cost gap is real but not dramatic, so cost alone isn't the strongest argument. (Plug in your own rates and call counts. Every extra step that carries the handbook multiplies these numbers.)

Recall. Long context is good now, but not perfect. Anthropic's launch numbers for Opus 4.6 on the 8-needle, 1M-token variant of MRCR v2 were around 76%. That's a synthetic benchmark at a much longer length than our handbook, so it says nothing precise about our documents. What it does say is that "the model will find the clause" is a claim you have to measure, and a miss on a financial contract is expensive.

Traceability. This is the argument that mattered most to us. With retrieval, the exact chunks that entered the prompt are a small, loggable, replayable artifact (exactly what a cut-point envelope wants). With a 175K-token dump, you can't tell which clause the model leaned on, and every edit to the handbook changes your entire prompt.

So we kept RAG. Our problem was that we were running 2023-era "RAG v1" in 2026.

Failure 1: Fixed-size chunking severed the condition from the formula

Our ingestion script split PDFs every 512 tokens. At 2:14 AM, the vector store held this:

chunk_041 (ended at token 512): "...For Enterprise Annual plans, seat reductions must be submitted in writing at least 30 calendar days prior to the renewal date. Mid-term or late seat reductions are non-refundable and apply only to the following renewal term."

chunk_042 (started at token 513): "Table 4.2 — Seat Downgrade Pro-Rata Credit Formula: credit_usd = (prior_seats - new_seats) * annual_seat_price * (days_remaining / 365)..."

The customer's message mentioned "downgraded 40 seats" and "billed", which matched the formula table well. chunk_042 scored 0.86 cosine similarity; chunk_041, the condition that made the formula inapplicable, scored 0.71 and missed the top-3 cutoff. The planner saw a formula with no precondition.

We made three changes:

Section-aware parent-child chunking. We index small child chunks for search but always return the full parent section (Section 4.2 and Table 4.2 together). Each document version is stamped with effective_from and superseded_at, so a stale policy can't be retrieved.
Hybrid search plus reranking. We pull the top 20 candidates using Reciprocal Rank Fusion over dense embeddings and BM25 keyword search, then pass them through a cross-encoder reranker to pick the top 5. On our 200-ticket regression suite, policy-exception recall went from 64% to 97%. (If you tune and report on the same suite, hold some tickets out.)
A regression test pinned to the incident. We reuse the harness from part 1, freeze the exact bad query rewrite from 2:14 AM, and check that the new retrieval still finds the notice clause:

def test_frozen_bad_rewrite_still_retrieves_notice_clause():
    session = CutPointSession(TRACE, mode="replay")
    bad_query = session.envelopes[("query_rewrite", 1)].output  # "...credit calculation table" (no "annual")

    chunks = retrieve(bad_query)  # new hybrid + rerank + parent-section pipeline

    assert any("30 calendar days" in c.text for c in chunks)

The upstream condition, a sampled rewrite that drops "annual", is still there. The pipeline is now robust to it.

Failure 2: Raw SQL over metrics with ambiguous meaning

At Boundary 3, the agent wrote SELECT active_seats FROM subscriptions against the warehouse through MCP. The query was valid. The problem was semantics: in our warehouse, active_seats is a live operational telemetry gauge, while contracted_billable_seats is the governed financial metric. The model picked a plausible column, and the nightly dbt job was mid-refresh, so it read 80 instead of 120.

Text-to-SQL was never the hard part; text-to-the-right-SQL is a semantics problem.

So we removed the run_sql() tool. The MCP server now exposes one governed tool that returns defined metrics in a single atomic read:

{
  "account_id": "acct_99201",
  "contracted_billable_seats": 120,
  "telemetry_active_seats": 80,
  "notice_window_days_remaining": -13
}

-13 is the whole story: the downgrade was requested 17 days before renewal, against a 30-day requirement. The agent no longer chooses between active_seats and contracted_billable_seats. The contract chooses, and the data comes from a consistent snapshot rather than a table mid-refresh.

With the full parent section and the governed metrics, the planner can't reach "automated overcharge error." And because the gate from part 1 sits at the exit, even a wrong planner decision can't move money.

Act IV: The latency and cost autopsy

Once the replay tests were green, we looked at why TICKET-8841 took 42.6 seconds and cost $0.184. The answer was embarrassing: we'd routed every boundary through a frontier reasoning model.

Asking a frontier model to rewrite a search query or format a JSON payload is paying for deliberation on a task that needs none. The planner decides whether the run succeeds, and the worker tier is close to interchangeable once the spec is unambiguous.

So we kept a frontier model on Boundary 4, the only step that needs multi-clause contract judgment, and routed the rest by task. I'm using model tiers instead of product names, because the rankings change monthly. Benchmark your own candidates against your own replay suite.

Boundary	Before	Before (latency / cost)	After	Why it fits	After (latency / cost)
1. PII scrub + query rewrite	Frontier reasoning model	6.4s / $0.022	Small, fast instruct model	Pattern extraction and rewriting need speed, not deep reasoning	0.35s / $0.0001
2. Retrieval	Dense vector search only	0.9s / $0.002	Hybrid + cross-encoder rerank, parent-section return	Keeps Section 4.2 and Table 4.2 together	1.15s / $0.0030
3. Seat lookup	Raw SQL written by a frontier model	7.8s / $0.031	Governed MCP tool, called by a mid-size tool-calling model	Structured call against a defined contract; no SQL authoring	0.90s / $0.0012
4. Policy adjudication	Frontier reasoning model	18.2s / $0.094	Same frontier model, second provider as failover	The one step that needs multi-clause contract judgment	2.80s / $0.0240 (trimmed to the parent section)
5. Ledger guard	Prompt-only check	4.1s / $0.015	Deterministic gate + human-approval interrupt	Hard invariants belong in code, not tokens	0.01s / $0.0000
6. Customer reply	Frontier reasoning model	5.2s / $0.020	Fast generation model	The decision is already locked as JSON; the reply is formatting	0.65s / $0.0035
Total per ticket		42.6s / $0.184			5.86s / $0.0318

End to end, latency fell 86% (42.6s to 5.86s) and cost per ticket fell 82.7%. At 45,000 tickets a month, that's about $8,280 versus $1,430 in inference, before the hosting cost of any self-hosted models.

Two things stand out in the table:

Correctness cost a little. Retrieval got slower and pricier (0.9s to 1.15s) because hybrid search and reranking do more work. That's the right trade.
The gate made routing safe. The "0 false approvals across 200 replay traces" result isn't something we get from the cheaper models behaving well. For the high-value path it's structural, because the gate blocks above $5,000 no matter what any model says. Routing experiments can't regress it, and accuracy parity on the rest of the suite is the thing we measured.

The pattern

Across both posts, the same shape keeps showing up. Probabilistic models are extraordinary at turning messy human intent into structured plans. They're bad at being their own database catalog, access-control layer, and regression suite. What worked was wrapping them in deterministic engineering:

Semantic contracts going in: parent-section retrieval and governed metric tools instead of raw SQL.
Surgical compute in the middle: one frontier call for the judgment step, cheap and fast workers elsewhere.
Hard invariants at the exit: money-movement gates in code, outside the agent.
Frozen cut-point replay in CI: so a 2:14 AM incident becomes a 14ms deterministic test forever.

If you're debugging an agent right now, try this first: pick your last real incident and ask whether your fix has ever been run against the exact upstream state that caused it. If the answer is "I re-ran it a few times," start there.

The Prompt Tweak That Fooled Us 5 Times, and the Code Gate That Actually Fixed It - Part 1

samarth — Tue, 20 Oct 2026 05:57:43 +0000

At 2:14 AM on a Tuesday, our billing reconciliation agent processed TICKET-8841. (Ticket IDs, table schemas, and exact dollar amounts below are reconstructed from production patterns so I don't leak our internal ledger.)

The customer's message was routine: "We downgraded 40 seats on March 15, before our April 1 annual renewal. Why were we billed for the full 120 seats?"

The agent had everything a modern 2026 AI stack is supposed to have: a flagship reasoning model (Claude Opus 4.6), billing policy chunks pulled from a vector database (RAG), and warehouse and Stripe ledger access through MCP (Model Context Protocol).

Forty-two seconds later, instead of drafting a credit memo for human approval, the agent executed:

{
  "tool": "issue_ledger_credit",
  "arguments": {
    "account_id": "acct_99201",
    "amount_usd": "48000.00",
    "auto_settle": true,
    "reason": "Pro-rata seat downgrade adjustment per Section 4.2"
  }
}

It settled a $48,000 credit for a customer whose contract explicitly locked seat counts 30 days before renewal.

But the $48,000 mistake isn't what this post is about. It's about what happened at 9:15 AM the next morning—because the way we initially "fixed" the bug is a trap teams fall into every day when debugging stochastic AI systems.

Act I: The 9:15 AM Prompt Tweak (And Why 5 Green Runs Lied)

At 9:15 AM, the on-call engineer opened the system prompt and added a bold rule:

CRITICAL: Never set auto_settle=True for credits above $5,000; always route to human_review.

Then we re-ran TICKET-8841 five times in staging. All five runs set auto_settle: false and routed the ticket to review. The PR was approved, merged, and deployed before lunch.

Seventy-two hours later, on TICKET-9102, the agent set auto_settle: true on a $31,500 mid-term downgrade and settled it again.

Why did five consecutive green test runs lie to us?

Because an agentic workflow isn't a pure function f(x) = y. It is a chain of stateful, sampled steps, and every step is a chance for the run to take a different path. When you re-run an agent from the beginning to verify a fix, you aren't replaying the incident—you are drawing one fresh sample from an environment that has already moved on.

Look at our billing agent's six-boundary execution graph:

[1. Query Rewrite] ──> [2. RAG Retrieval] ──> [3. Warehouse MCP SQL]
                                                       │
[6. User Reply] <── [5. Ledger MCP Tool] <── [4. Policy Reasoning Planner]

When we audited the raw 2:14 AM logs, the model hadn't simply ignored our system prompt. Three upstream conditions collided simultaneously:

Boundaries 1–2 (Rewrite + Retrieval): The query rewriter sampled "seat downgrade credit calculation table", dropping the word "annual". Because retrieval is deterministic given the query, it returned chunk_042 (the pro-rata formula table) without chunk_041 (the preceding paragraph requiring 30 days' notice on annual plans). Treating the rewrite-retrieval pair as one event, that omission happened in roughly 1 of 4 runs (p ≈ 0.25 at temperature=0.3).
Boundary 3 (Warehouse MCP SQL): The agent ran SELECT active_seats FROM subscriptions at 2:14 AM, while a nightly dbt batch job was mid-refresh. It read 80 instead of the contracted 120 (p ≈ 0.12 across our 24-hour ticket volume during batch windows).
Boundary 4 (Policy Planner): Only with the notice clause missing and the mismatched seat count did the planner conclude the charge was an "automated billing overcharge error" instead of a "policy exception." And an older paragraph in our system prompt explicitly exempted "automated billing overcharge errors" from the $5,000 review cap.

Now look at the math of our "five green re-runs" at 9:15 AM:

At 9:15 AM, the nightly dbt batch had finished hours earlier. The mid-refresh warehouse state was impossible (p = 0), meaning the buggy branch was completely unreachable. Five green checks were guaranteed: a 100% false-pass rate. (If you let upstream tool calls hit a live database that has moved on since the incident, your replay leaks reality.)

Even if we had tested against a chaos staging environment where that database state recurred randomly 12% of the time:

P(enter buggy branch, per run) = 0.25 × 0.12 = 0.03
P(5 independent runs miss it)  = (1 - 0.03)^5 = 0.97^5 ≈ 0.859

That is an 85.9% chance of seeing five green checks without your prompt change ever being tested against the state that caused the outage. And in practice it's even worse: five re-runs executed thirty seconds apart share the same underlying database snapshot, so they aren't five independent draws—they're one draw repeated five times.

Act II: Proving the Fix With Name-Scoped Cut-Point Replay

The iron law of agent debugging is simple: freeze the run, not the model.

Instead of re-running Boundaries 1, 2, and 3 live, you record every boundary's exact input/output envelope in production. To verify a fix, you freeze everything upstream of the boundary you changed, drop your new code into the middle of the frozen 2:14 AM incident state, and run live only from that cut-point.

A Trace-Indexing Bug That Bit Us

Our first internal replay wrapper keyed recorded envelopes by flat execution order (step_0, step_1, step_2, ...). We immediately hit an edge case: what happens when your fix adds a retry or an extra validation call?

If Boundary 3 (warehouse_mcp_sql) retries once on a timeout, the retry becomes step_3, shifting the old step_3 (policy_planner_llm) to step_4. Every stubbed boundary after the retry silently reads the wrong recorded envelope or crashes on a schema mismatch.

The fix is to key every envelope by (boundary_name, per_name_call_index) rather than flat position, and to scrub PII and secrets before the envelope touches disk so incident traces can be committed directly to Git as CI fixtures.

Here is a complete, self-contained 90-line implementation you can drop into pytest today (or if you want a full-featured library with graph lineage, use pip install agent-chronicle):

import json
import re
from collections import defaultdict
from dataclasses import asdict, dataclass
from decimal import Decimal
from functools import wraps
from pathlib import Path
from typing import Any, Callable

PII_PATTERNS = {
    r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b": "<REDACTED_EMAIL>",
    r"\bsk-[A-Za-z0-9]{20,}\b": "<REDACTED_API_KEY>",
}


def redact(obj: Any) -> Any:
    """Scrub PII and secrets before any trace envelope touches disk."""
    raw = json.dumps(obj, sort_keys=True, default=str)
    for pattern, token in PII_PATTERNS.items():
        raw = re.sub(pattern, token, raw)
    return json.loads(raw)


@dataclass
class Envelope:
    boundary: str
    call_index: int
    inputs: dict[str, Any]
    output: Any
    model_version: str


class CutPointSession:
    def __init__(self, trace_path: Path, mode: str = "record"):
        assert mode in ("record", "replay")
        self.trace_path = trace_path
        self.mode = mode
        self.counters: dict[str, int] = defaultdict(int)
        self.stubbed: set[tuple[str, int]] = set()
        self.envelopes: dict[tuple[str, int], Envelope] = {}
        self.captured_live: dict[tuple[str, int], Any] = {}

        if mode == "replay":
            for line in trace_path.read_text().splitlines():
                env = Envelope(**json.loads(line))
                self.envelopes[(env.boundary, env.call_index)] = env

    def stub(self, boundary: str, call_index: int = 1) -> "CutPointSession":
        """Serve the frozen 2:14 AM output for this boundary call."""
        if (boundary, call_index) not in self.envelopes:
            raise KeyError(f"No recorded envelope for {boundary}#{call_index}")
        self.stubbed.add((boundary, call_index))
        return self

    def wrap(self, boundary: str, model_version: str = "deterministic-tool"):
        def decorator(fn: Callable) -> Callable:
            @wraps(fn)
            def wrapper(*args, **kwargs):
                self.counters[boundary] += 1
                key = (boundary, self.counters[boundary])

                if self.mode == "replay":
                    if key in self.stubbed:
                        return self.envelopes[key].output
                    result = fn(*args, **kwargs)
                    self.captured_live[key] = redact(result)
                    return result

                result = fn(*args, **kwargs)
                env = Envelope(
                    boundary=boundary,
                    call_index=key[1],
                    inputs=redact({"args": args, "kwargs": kwargs}),
                    output=redact(result),
                    model_version=model_version,
                )
                with self.trace_path.open("a") as f:
                    f.write(json.dumps(asdict(env)) + "\n")
                return result

            return wrapper

        return decorator

Because each boundary tracks its own counter, if a fix adds a retry to warehouse_mcp_sql, that retry is indexed as ("warehouse_mcp_sql", 2). Meanwhile ("policy_planner_llm", 1) keeps its exact identity. Anything not explicitly stubbed runs live.

Running Two Experiments Against the Frozen 2:14 AM Incident

Once fixtures/incident_8841.jsonl is committed to the repo, we can separate "Did the prompt tweak change the model's mind?" from "Can the system ever wire $48,000 without approval again?"

TRACE = Path("fixtures/incident_8841.jsonl")

def replay_session(stubbed_boundaries: list[str]) -> CutPointSession:
    session = CutPointSession(TRACE, mode="replay")
    for name in stubbed_boundaries:
        session.stub(name, 1)
    return session

Experiment A: Testing the 9:15 AM Prompt Tweak (Planner Live)

We freeze Boundaries 1–3 (query_rewrite, rag_retrieve, and warehouse_mcp_sql) so the planner receives the exact severed chunk and mid-refresh 80-seat count from 2:14 AM, and we run policy_planner_llm live 10 times with the new "CRITICAL: Never set auto_settle=True..." prompt:

auto_settle_flags = []
for _ in range(10):
    session = replay_session(["query_rewrite", "rag_retrieve", "warehouse_mcp_sql"])
    run_ticket_workflow(session, ticket_id="TICKET-8841", dry_run_stripe=True)
    decision = session.captured_live[("policy_planner_llm", 1)]
    auto_settle_flags.append(decision["auto_settle"])

print(f"Prompt-only fix still set auto_settle=True in {sum(auto_settle_flags)}/10 runs")
# Output: Prompt-only fix still set auto_settle=True in 4/10 runs

When forced to confront the exact conflicting inputs from 2:14 AM, the model still reasoned that an "automated overcharge error" overruled the $5,000 threshold 4 times out of 10. Our 9:15 AM prompt fix was a placebo.

Experiment B: Moving the Invariant Into Code (Planner Stubbed, Tool Live)

Instead of begging the model to respect a dollar threshold in prose, we enforce the threshold deterministically inside the tool boundary (or via a LangGraph human-in-the-loop checkpoint interrupt):

SINGLE_CREDIT_CAP_USD = Decimal("5000.00")
DAILY_ACCOUNT_CAP_USD = Decimal("5000.00")


def build_ledger_tool(session: CutPointSession, ledger_Gateway):
    @session.wrap("issue_ledger_credit")
    def issue_ledger_credit(
        account_id: str, amount_usd: str | float, auto_settle: bool, reason: str
    ) -> dict:
        amount = Decimal(str(amount_usd))  # Never compare money as IEEE-754 floats
        settled_24h = ledger_Gateway.get_settled_credits_24h(account_id)

        # Hard invariant: cap both single-credit and rolling 24h exposure
        if amount > SINGLE_CREDIT_CAP_USD or (settled_24h + amount) > DAILY_ACCOUNT_CAP_USD:
            return {
                "status": "blocked_pending_human_approval",
                "blocked": True,
                "required_approver_role": "finance_controller",
                "proposed_amount_usd": str(amount),
            }
        if not auto_settle:
            return {"status": "drafted_for_review", "blocked": False}
        return ledger_Gateway.execute_stripe_transfer(account_id, amount, reason)

    return issue_ledger_credit

Notice two details in that gate:

It ignores auto_settle whenever the cap is exceeded. The model's boolean flag is treated as an untrusted suggestion, never an authorization.
It checks settled_24h + amount. A per-call cap alone can be bypassed by an agent loop issuing ten sequential $4,900 credits.

Now we stub all four upstream boundaries—including policy_planner_llm—so the exact bad decision from 2:14 AM (auto_settle=True, amount_usd="48000.00") is replayed into our live tool boundary with zero LLM calls:

def test_incident_8841_cannot_move_money(mock_ledger_gateway):
    session = replay_session([
        "query_rewrite",
        "rag_retrieve",
        "warehouse_mcp_sql",
        "policy_planner_llm",
    ])
    issue_credit = build_ledger_tool(session, mock_ledger_gateway)

    # Replay the exact planner payload captured at 2:14 AM
    bad_plan = session.envelopes[("policy_planner_llm", 1)].output
    result = issue_credit(**bad_plan["arguments"])

    assert result["blocked"] is True
    assert session.captured_live[("issue_ledger_credit", 1)]["status"] == "blocked_pending_human_approval"
    mock_ledger_gateway.execute_stripe_transfer.assert_not_called()

That test runs in ~14 milliseconds in CI with no API keys, no network calls, and zero flakiness.

What the Gate Didn't Fix

Putting a deterministic gate at Boundary 5 contained the blast radius, but it didn't fix the upstream brain damage: why did our RAG pipeline sever the 30-day notice requirement from the refund formula, why did our MCP server let the agent query the wrong seats column, and why was a 6-step ticket taking 42.6 seconds?

We'll tear into those upstream bugs—and the step-wise routing table that cut our latency by 86%—in Part 2.

Takeaways From Part 1

A green re-run is one sample, not a proof: If your live environment can no longer reach the 2:14 AM state, five green re-runs give you a 100% false-pass rate.
Separate behavioral evals from safety gates: Run the planner live over frozen upstream inputs (Experiment A) to measure prompt/model improvements, but stub the planner (Experiment B) to prove your code invariants hold even when the model hallucinates.
Key traces by (boundary_name, call_index): Flat step indexing breaks the moment a fix introduces a retry.
Law outranks intelligence: Prompts are wishes; money movement belongs in deterministic code.