The Prompt Tweak That Fooled Us 5 Times, and the Code Gate That Actually Fixed It - Part 1

At 2:14 AM on a Tuesday, our billing reconciliation agent processed TICKET-8841. (Ticket IDs, table schemas, and exact dollar amounts below are reconstructed from production patterns so I don't leak our internal ledger.)

The customer's message was routine: "We downgraded 40 seats on March 15, before our April 1 annual renewal. Why were we billed for the full 120 seats?"

The agent had everything a modern 2026 AI stack is supposed to have: a flagship reasoning model (Claude Opus 4.6), billing policy chunks pulled from a vector database (RAG), and warehouse and Stripe ledger access through MCP (Model Context Protocol).

Forty-two seconds later, instead of drafting a credit memo for human approval, the agent executed:

{
  "tool": "issue_ledger_credit",
  "arguments": {
    "account_id": "acct_99201",
    "amount_usd": "48000.00",
    "auto_settle": true,
    "reason": "Pro-rata seat downgrade adjustment per Section 4.2"
  }
}

It settled a $48,000 credit for a customer whose contract explicitly locked seat counts 30 days before renewal.

But the $48,000 mistake isn't what this post is about. It's about what happened at 9:15 AM the next morning—because the way we initially "fixed" the bug is a trap teams fall into every day when debugging stochastic AI systems.

Act I: The 9:15 AM Prompt Tweak (And Why 5 Green Runs Lied)

At 9:15 AM, the on-call engineer opened the system prompt and added a bold rule:

CRITICAL: Never set auto_settle=True for credits above $5,000; always route to human_review.

Then we re-ran TICKET-8841 five times in staging. All five runs set auto_settle: false and routed the ticket to review. The PR was approved, merged, and deployed before lunch.

Seventy-two hours later, on TICKET-9102, the agent set auto_settle: true on a $31,500 mid-term downgrade and settled it again.

Why did five consecutive green test runs lie to us?

Because an agentic workflow isn't a pure function f(x) = y. It is a chain of stateful, sampled steps, and every step is a chance for the run to take a different path. When you re-run an agent from the beginning to verify a fix, you aren't replaying the incident—you are drawing one fresh sample from an environment that has already moved on.

Look at our billing agent's six-boundary execution graph:

[1. Query Rewrite] ──> [2. RAG Retrieval] ──> [3. Warehouse MCP SQL]
                                                       │
[6. User Reply] <── [5. Ledger MCP Tool] <── [4. Policy Reasoning Planner]

When we audited the raw 2:14 AM logs, the model hadn't simply ignored our system prompt. Three upstream conditions collided simultaneously:

1. Boundaries 1–2 (Rewrite + Retrieval): The query rewriter sampled "seat downgrade credit calculation table", dropping the word "annual". Because retrieval is deterministic given the query, it returned chunk_042 (the pro-rata formula table) without chunk_041 (the preceding paragraph requiring 30 days' notice on annual plans). Treating the rewrite-retrieval pair as one event, that omission happened in roughly 1 of 4 runs (p ≈ 0.25 at temperature=0.3).
2. Boundary 3 (Warehouse MCP SQL): The agent ran SELECT active_seats FROM subscriptions at 2:14 AM, while a nightly dbt batch job was mid-refresh. It read 80 instead of the contracted 120 (p ≈ 0.12 across our 24-hour ticket volume during batch windows).
3. Boundary 4 (Policy Planner): Only with the notice clause missing and the mismatched seat count did the planner conclude the charge was an "automated billing overcharge error" instead of a "policy exception." And an older paragraph in our system prompt explicitly exempted "automated billing overcharge errors" from the $5,000 review cap.

Now look at the math of our "five green re-runs" at 9:15 AM:

At 9:15 AM, the nightly dbt batch had finished hours earlier. The mid-refresh warehouse state was impossible (p = 0), meaning the buggy branch was completely unreachable. Five green checks were guaranteed: a 100% false-pass rate. (If you let upstream tool calls hit a live database that has moved on since the incident, your replay leaks reality.)

Even if we had tested against a chaos staging environment where that database state recurred randomly 12% of the time:

P(enter buggy branch, per run) = 0.25 × 0.12 = 0.03
P(5 independent runs miss it)  = (1 - 0.03)^5 = 0.97^5 ≈ 0.859

That is an 85.9% chance of seeing five green checks without your prompt change ever being tested against the state that caused the outage. And in practice it's even worse: five re-runs executed thirty seconds apart share the same underlying database snapshot, so they aren't five independent draws—they're one draw repeated five times.

Act II: Proving the Fix With Name-Scoped Cut-Point Replay

The iron law of agent debugging is simple: freeze the run, not the model.

Instead of re-running Boundaries 1, 2, and 3 live, you record every boundary's exact input/output envelope in production. To verify a fix, you freeze everything upstream of the boundary you changed, drop your new code into the middle of the frozen 2:14 AM incident state, and run live only from that cut-point.

A Trace-Indexing Bug That Bit Us

Our first internal replay wrapper keyed recorded envelopes by flat execution order (step_0, step_1, step_2, ...). We immediately hit an edge case: what happens when your fix adds a retry or an extra validation call?

If Boundary 3 (warehouse_mcp_sql) retries once on a timeout, the retry becomes step_3, shifting the old step_3 (policy_planner_llm) to step_4. Every stubbed boundary after the retry silently reads the wrong recorded envelope or crashes on a schema mismatch.

The fix is to key every envelope by (boundary_name, per_name_call_index) rather than flat position, and to scrub PII and secrets before the envelope touches disk so incident traces can be committed directly to Git as CI fixtures.

Here is a complete, self-contained 90-line implementation you can drop into pytest today (or if you want a full-featured library with graph lineage, use pip install agent-chronicle):

import json
import re
from collections import defaultdict
from dataclasses import asdict, dataclass
from decimal import Decimal
from functools import wraps
from pathlib import Path
from typing import Any, Callable

PII_PATTERNS = {
    r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b": "<REDACTED_EMAIL>",
    r"\bsk-[A-Za-z0-9]{20,}\b": "<REDACTED_API_KEY>",
}


def redact(obj: Any) -> Any:
    """Scrub PII and secrets before any trace envelope touches disk."""
    raw = json.dumps(obj, sort_keys=True, default=str)
    for pattern, token in PII_PATTERNS.items():
        raw = re.sub(pattern, token, raw)
    return json.loads(raw)


@dataclass
class Envelope:
    boundary: str
    call_index: int
    inputs: dict[str, Any]
    output: Any
    model_version: str


class CutPointSession:
    def __init__(self, trace_path: Path, mode: str = "record"):
        assert mode in ("record", "replay")
        self.trace_path = trace_path
        self.mode = mode
        self.counters: dict[str, int] = defaultdict(int)
        self.stubbed: set[tuple[str, int]] = set()
        self.envelopes: dict[tuple[str, int], Envelope] = {}
        self.captured_live: dict[tuple[str, int], Any] = {}

        if mode == "replay":
            for line in trace_path.read_text().splitlines():
                env = Envelope(**json.loads(line))
                self.envelopes[(env.boundary, env.call_index)] = env

    def stub(self, boundary: str, call_index: int = 1) -> "CutPointSession":
        """Serve the frozen 2:14 AM output for this boundary call."""
        if (boundary, call_index) not in self.envelopes:
            raise KeyError(f"No recorded envelope for {boundary}#{call_index}")
        self.stubbed.add((boundary, call_index))
        return self

    def wrap(self, boundary: str, model_version: str = "deterministic-tool"):
        def decorator(fn: Callable) -> Callable:
            @wraps(fn)
            def wrapper(*args, **kwargs):
                self.counters[boundary] += 1
                key = (boundary, self.counters[boundary])

                if self.mode == "replay":
                    if key in self.stubbed:
                        return self.envelopes[key].output
                    result = fn(*args, **kwargs)
                    self.captured_live[key] = redact(result)
                    return result

                result = fn(*args, **kwargs)
                env = Envelope(
                    boundary=boundary,
                    call_index=key[1],
                    inputs=redact({"args": args, "kwargs": kwargs}),
                    output=redact(result),
                    model_version=model_version,
                )
                with self.trace_path.open("a") as f:
                    f.write(json.dumps(asdict(env)) + "\n")
                return result

            return wrapper

        return decorator

Because each boundary tracks its own counter, if a fix adds a retry to warehouse_mcp_sql, that retry is indexed as ("warehouse_mcp_sql", 2). Meanwhile ("policy_planner_llm", 1) keeps its exact identity. Anything not explicitly stubbed runs live.

Running Two Experiments Against the Frozen 2:14 AM Incident

Once fixtures/incident_8841.jsonl is committed to the repo, we can separate "Did the prompt tweak change the model's mind?" from "Can the system ever wire $48,000 without approval again?"

TRACE = Path("fixtures/incident_8841.jsonl")

def replay_session(stubbed_boundaries: list[str]) -> CutPointSession:
    session = CutPointSession(TRACE, mode="replay")
    for name in stubbed_boundaries:
        session.stub(name, 1)
    return session

Experiment A: Testing the 9:15 AM Prompt Tweak (Planner Live)

We freeze Boundaries 1–3 (query_rewrite, rag_retrieve, and warehouse_mcp_sql) so the planner receives the exact severed chunk and mid-refresh 80-seat count from 2:14 AM, and we run policy_planner_llm live 10 times with the new "CRITICAL: Never set auto_settle=True..." prompt:

auto_settle_flags = []
for _ in range(10):
    session = replay_session(["query_rewrite", "rag_retrieve", "warehouse_mcp_sql"])
    run_ticket_workflow(session, ticket_id="TICKET-8841", dry_run_stripe=True)
    decision = session.captured_live[("policy_planner_llm", 1)]
    auto_settle_flags.append(decision["auto_settle"])

print(f"Prompt-only fix still set auto_settle=True in {sum(auto_settle_flags)}/10 runs")
# Output: Prompt-only fix still set auto_settle=True in 4/10 runs

When forced to confront the exact conflicting inputs from 2:14 AM, the model still reasoned that an "automated overcharge error" overruled the $5,000 threshold 4 times out of 10. Our 9:15 AM prompt fix was a placebo.

Experiment B: Moving the Invariant Into Code (Planner Stubbed, Tool Live)

Instead of begging the model to respect a dollar threshold in prose, we enforce the threshold deterministically inside the tool boundary (or via a LangGraph human-in-the-loop checkpoint interrupt):

SINGLE_CREDIT_CAP_USD = Decimal("5000.00")
DAILY_ACCOUNT_CAP_USD = Decimal("5000.00")


def build_ledger_tool(session: CutPointSession, ledger_Gateway):
    @session.wrap("issue_ledger_credit")
    def issue_ledger_credit(
        account_id: str, amount_usd: str | float, auto_settle: bool, reason: str
    ) -> dict:
        amount = Decimal(str(amount_usd))  # Never compare money as IEEE-754 floats
        settled_24h = ledger_Gateway.get_settled_credits_24h(account_id)

        # Hard invariant: cap both single-credit and rolling 24h exposure
        if amount > SINGLE_CREDIT_CAP_USD or (settled_24h + amount) > DAILY_ACCOUNT_CAP_USD:
            return {
                "status": "blocked_pending_human_approval",
                "blocked": True,
                "required_approver_role": "finance_controller",
                "proposed_amount_usd": str(amount),
            }
        if not auto_settle:
            return {"status": "drafted_for_review", "blocked": False}
        return ledger_Gateway.execute_stripe_transfer(account_id, amount, reason)

    return issue_ledger_credit

Notice two details in that gate:

1. It ignores auto_settle whenever the cap is exceeded. The model's boolean flag is treated as an untrusted suggestion, never an authorization.
2. It checks settled_24h + amount. A per-call cap alone can be bypassed by an agent loop issuing ten sequential $4,900 credits.

Now we stub all four upstream boundaries—including policy_planner_llm—so the exact bad decision from 2:14 AM (auto_settle=True, amount_usd="48000.00") is replayed into our live tool boundary with zero LLM calls:

def test_incident_8841_cannot_move_money(mock_ledger_gateway):
    session = replay_session([
        "query_rewrite",
        "rag_retrieve",
        "warehouse_mcp_sql",
        "policy_planner_llm",
    ])
    issue_credit = build_ledger_tool(session, mock_ledger_gateway)

    # Replay the exact planner payload captured at 2:14 AM
    bad_plan = session.envelopes[("policy_planner_llm", 1)].output
    result = issue_credit(**bad_plan["arguments"])

    assert result["blocked"] is True
    assert session.captured_live[("issue_ledger_credit", 1)]["status"] == "blocked_pending_human_approval"
    mock_ledger_gateway.execute_stripe_transfer.assert_not_called()

That test runs in ~14 milliseconds in CI with no API keys, no network calls, and zero flakiness.

What the Gate Didn't Fix

Putting a deterministic gate at Boundary 5 contained the blast radius, but it didn't fix the upstream brain damage: why did our RAG pipeline sever the 30-day notice requirement from the refund formula, why did our MCP server let the agent query the wrong seats column, and why was a 6-step ticket taking 42.6 seconds?

We'll tear into those upstream bugs—and the step-wise routing table that cut our latency by 86%—in Part 2.

Takeaways From Part 1

• A green re-run is one sample, not a proof: If your live environment can no longer reach the 2:14 AM state, five green re-runs give you a 100% false-pass rate.
• Separate behavioral evals from safety gates: Run the planner live over frozen upstream inputs (Experiment A) to measure prompt/model improvements, but stub the planner (Experiment B) to prove your code invariants hold even when the model hallucinates.
• Key traces by (boundary_name, call_index): Flat step indexing breaks the moment a fix introduces a retry.
• Law outranks intelligence: Prompts are wishes; money movement belongs in deterministic code.