DEV Community: James O'Connor

# The Partial JSON Looked Done. It Wasn't. Here's What Streaming Structured Output Actually Requires.

James O'Connor — Mon, 06 Jul 2026 00:40:29 +0000

Last quarter we shipped a contract extraction feature that streamed its output field by field into a review UI, so a paralegal could start reading the extracted renewal date and counterparty name before the model finished the whole document. Nice idea. Faster perceived latency, better demo, the kind of thing that gets a thumbs up in a design review.

Then a reviewer flagged a contract where the counterparty name in the UI read "Meridian Hold" and the final, settled value was "Meridian Holdings Group LLC." Nobody had touched anything. The UI had simply displayed the string as it existed at a moment mid-stream, before the model had finished writing it, and the user had already glanced at it, nodded, and moved to the next tab. By the time the full value arrived, the case for that field was already closed in the reviewer's head.

That's the part that stings about streaming structured output. It doesn't usually fail loud. It fails by being plausible at exactly the wrong instant.

The naive approach is to accumulate the streamed text and try json.loads on the buffer every time a new chunk lands, catching the exception when it's not valid yet. That works, technically, right up until it doesn't, and the failure mode is worse than a crash: it's a false negative that becomes a false positive.

Concretely, here's what breaks:

1. Strings that parse as valid but aren't finished. json.loads doesn't fail on {"counterparty": "Meridian Hold"}. That's completely valid JSON. It's also not the value you want. A string field only becomes trustworthy at the character where its closing quote lands, and nothing before that tells you where that is, because the model doesn't announce "I'm two tokens from done."

2. Arrays that look closed but aren't. With tool-calling providers that stream token-by-token, we've seen an array of extracted clauses close its bracket, then reopen because the provider's underlying generation retried a truncated chunk server-side (this is provider-side behavior we can't fully control or always observe, and it showed up more under load, though I'd want a longer sample before I called that a hard rule rather than a pattern we noticed on high-traffic days). If your parser saw the first closing bracket and moved on, you've committed to a value that got silently superseded.

3. Parallel tool calls interleaving. When a model issues more than one tool call in the same turn, providers don't always guarantee the chunks for call A finish before chunks for call B start arriving. We had a case where two extract_clause calls were in flight and a naive buffer-per-response (rather than buffer-per-call) approach spliced fragments from both into one JSON blob that parsed successfully and meant nothing.

None of these are edge cases in the sense of being rare. In our sample of roughly 400 flagged extraction sessions over about six weeks (anecdotal, drawn from our own error queue, not an industry number), something in this family, a field displayed before it was complete, showed up in just under 3% of streamed sessions. Low frequency, high cost, because the ones that go wrong are the ones a human trusted.

What "schema-aware" actually means here

The fix isn't a smarter JSON parser. Full-document partial-JSON parsers exist and are useful for a different problem (rendering a tree view of an in-progress object), but they answer the wrong question. The question isn't "can I parse this yet." It's "is this specific field's value done being written."

That reframes the problem as a cursor per field, not a parser over the whole buffer. You track, for each key you care about, whether you've seen its terminating character. For strings, that's an unescaped closing quote. You do not surface the field to any downstream consumer, UI or otherwise, until that condition is met. Everything before that point stays internal state, not output.

Here's a minimal version of that, enough to show the shape (not a production parser, we handle nested objects and arrays with a similar but longer state machine):

from typing import Optional


class FieldExtractor:
    """Extracts only fields whose values are structurally complete
    from a growing buffer of streamed JSON text. Never calls
    json.loads on the partial buffer.

    A field counts as 'safe to surface' only once we've seen the
    terminating character for its type: here, a closing quote for
    strings (not preceded by an odd number of backslashes).
    """

    def __init__(self, expected_keys: set):
        self.expected_keys = expected_keys
        self.raw = ""
        self.emitted = {}

    def feed(self, chunk: str) -> dict:
        self.raw += chunk
        newly_closed = {}
        for key in self.expected_keys:
            if key in self.emitted:
                continue
            value = self._extract_closed_string_value(key)
            if value is not None:
                self.emitted[key] = value
                newly_closed[key] = value
        return newly_closed

    def _extract_closed_string_value(self, key: str) -> Optional[str]:
        marker = f'"{key}"'
        start = self.raw.find(marker)
        if start == -1:
            return None
        after_colon = self.raw.find(":", start + len(marker))
        if after_colon == -1:
            return None
        i = after_colon + 1
        while i < len(self.raw) and self.raw[i] in " \t\n":
            i += 1
        if i >= len(self.raw) or self.raw[i] != '"':
            return None  # not a string field, or value hasn't started
        j = i + 1
        while j < len(self.raw):
            if self.raw[j] == '"' and self.raw[j - 1] != "\\":
                return self.raw[i + 1:j]  # closing quote found, value is safe
            j += 1
        return None  # still being written


if __name__ == "__main__":
    extractor = FieldExtractor(expected_keys={"vendor_name", "renewal_date"})
    stream_chunks = [
        '{"vendor_name": "Acme Ind',
        'ustries, Inc.", "renewal_date": "2027-0',
        '3-01"}',
    ]
    for chunk in stream_chunks:
        closed = extractor.feed(chunk)
        for k, v in closed.items():
            print(f"safe to show: {k} = {v!r}")

Run it and the first chunk produces nothing (the vendor name string hasn't closed), the second chunk produces vendor_name, and the third produces renewal_date. Nothing gets displayed until its own value is structurally complete, independent of whatever else is still being written elsewhere in the object.

The version we actually run in production extends this with a stack for nested objects and arrays (so a clause list only emits an item once that item's closing brace is seen, not when the array itself closes), and a separate buffer keyed by tool-call ID so parallel calls can't splice into each other. Pydantic still validates the finished object at the end, same as before. This layer sits earlier and answers a narrower question: not "is this valid," but "is this done."

The part that's easy to miss

The instinct once you've built something like this is to treat it as purely a UI nicety, debounce the flicker, smooth the experience. But the actual failure we hit wasn't a UI glitch. It was an epistemic one: a human formed a belief about a fact ("counterparty is Meridian Hold, whatever that is") from data that hadn't finished existing yet. The fix isn't cosmetic. It's a correctness boundary between "data that exists" and "data that is still in the process of becoming data," and once you see it that way, treating it as a parsing convenience undersells what's actually broken.

Where I'd push back on this

The strongest objection to all of the above is that schema-aware incremental parsing adds real complexity, a stateful cursor per field, careful handling of escape sequences, tool-call ID bookkeeping, for a problem that a simpler mitigation might solve just as well: don't stream field values into a UI at all, stream only a coarse progress indicator ("extracting... 60%"), and reveal the full object once json.loads succeeds on the complete response. That removes the entire failure class in one move, and for a lot of products that's the right call, especially early on, when the team building the UI doesn't want to own a state machine.

I'd accept that objection for a lot of use cases. Where I wouldn't accept it is anywhere the whole point of streaming was the perceived-latency win, which was our actual reason for building this. If the product requirement is "show the paralegal something before the model finishes," you've already decided you need partial data, and the choice is really between partial data that's honest about its own completeness and partial data that isn't. Once you frame it that way, the extra state tracking looks less like gold-plating and more like the minimum bar for showing someone a fact you're asking them to trust.

The other pushback worth taking seriously: none of this catches a value that's structurally complete but semantically wrong, a well-formed string that just happens to be the wrong string. That's a different failure class with a different fix, and conflating the two is its own mistake.

Structured output broke on us three times. The third time taught us operator-ready.

James O'Connor — Fri, 03 Jul 2026 00:32:22 +0000

Structured output broke on us three times. The third time taught us what "operator-ready" means.

Last quarter we shipped a contract-extraction agent to an enterprise legal team. Schema validation passing at 97%. Human reviewers satisfied with the output quality in testing. Rollout went smoothly.

Then it broke. Three times. In three completely different ways.

The first two failures we fixed with better prompts and stricter schemas. The third one taught us something the first two hadn't: that "operator-ready" is not a technical checklist. It's a claim about your agent's behavior under conditions you didn't design it for.

Failure one: the validation paradox

Week two. A lease agreement came through with a renewal clause formatted as a table instead of prose. Our extractor looked for renewal terms in a specific JSON path. The table format populated the schema differently. Validation passed. The extracted renewal date was off by two years.

The fix was obvious in retrospect: add a canonical-format normalization step before extraction. But the lesson was sharper than that.

Schema validation tells you the shape of the output, not whether the content is correct. A JSON object with the right keys and the right types can still contain wrong values. Our 97% validation success rate was measuring the wrong thing. It was measuring structure conformance, not content accuracy.

After this failure, we separated validation into two signals: schema validity (does the object have the required fields) and field confidence (do we have evidence the content is correct). We started logging both. An output is trusted only when both signals are above threshold.

Failure two: the retry loop that lies

Month one. A particular clause type appeared in a contract format we hadn't trained our test set on. The extractor failed schema validation on the first attempt. Our retry logic kicked in, filled missing fields with model-inferred defaults, and passed validation on the third try.

The output looked right. The content was wrong. The inferred defaults were plausible values that did not match the actual contract.

No alert fired. No human review was triggered. The error surfaced three weeks later when the legal team flagged a discrepancy in a signed agreement.

This is the retry paradox: the retry loop is supposed to handle uncertainty, but in practice it converts "the model doesn't know" into "the model confidently guessed." The schema never sees the difference.

The fix: when a retry fails because of missing content (not format), the correct behavior is a human-review flag, not a default fill. "I cannot extract this clause with confidence" is a better output than a wrong value that passes validation.

We changed the retry logic to distinguish format failures (retry and reformat) from content failures (flag for review). The human-review rate went up. The silent error rate went to zero.

Failure three: the operator's data

This one took longer to understand.

Six weeks in, a new batch of contracts arrived from a subsidiary the legal team had recently acquired. Different contract structure, different clause naming conventions, different language patterns. Our extraction accuracy dropped from 94% on the training-corpus contracts to 61% on the acquired subsidiary's contracts.

We had not seen a single document from that subsidiary during development. Neither had our test suite.

This is the distribution shift problem. And it is the actual definition of not-operator-ready.

Production-ready means your agent handles the inputs you tested it on. Operator-ready means your agent handles the inputs the operator is actually going to give it. Those are not the same set.

The fix was not a better model or a better prompt. It was a process change: before any operator handoff, run the agent on a sample of the operator's own documents, measure accuracy on that corpus specifically, and establish a baseline before you commit to SLA numbers.

We now require 50 documents from the operator's corpus as part of the pre-handoff checklist. Not synthetic. Not ours. Theirs. If the accuracy on those 50 documents is not close to the accuracy on our training corpus, the handoff gets delayed until we understand why.

What these three failures have in common

All three were invisible to our eval suite. All three were visible with the right diagnostic.

The pattern: our eval was measuring our best case (our data, our test set, our format assumptions). Operator-ready means measuring the operator's case. Those are different measurement problems.

The three things we added to our pre-handoff process:

Field-level confidence scoring on every output (not just schema validity)
Content-failure-vs-format-failure separation in retry logic (fail loudly, not silently)
Operator corpus sampling before go-live (50 documents from their actual data, reviewed manually)

None of these are in the standard "production-ready" checklist. They're in the operator-ready checklist.

Where I'd push back on this

The common response to these failures is "just add more training data" or "fine-tune on the operator's corpus." That's the right long-term fix. It's not the short-term answer.

Fine-tuning takes weeks and requires labeling budget. An operator pilot that's already started does not have that runway. The faster path is: understand the distribution shift before you commit to accuracy numbers, not after you've already missed them.

There's also a steelman for the current "validation is enough" approach: for low-stakes use cases with structured, predictable inputs, schema validation really is sufficient. If every contract you're extracting is from the same template, format conformance and content accuracy are highly correlated.

The problem is that enterprise operators rarely have one template. The legal team that deployed our extractor manages contracts from 14 different counterparties, each with their own conventions. Validation-only was always going to break.

The concession I'll make: this is a data problem as much as an engineering problem. The teams that invest in building labeled corpora per operator will have substantially better outcomes than the teams that treat operator-ready as a single deployment decision. We didn't invest in that early enough. The second and third failures were partly the cost of that.

Operator-ready is not a state you reach. It's a process you run.

# Evaluating an AI agent is not evaluating an LLM call:

James O'Connor — Sun, 28 Jun 2026 22:58:50 +0000

I compared six tools for evaluating AI agents: LangSmith, Galileo, Arize Phoenix, Braintrust, Future AGI, and Langfuse. My thesis, up front so you can argue with it early: the mistake that wastes the most time is grading the agent's final answer like it is a single LLM call. An agent has a trajectory, which tools it called, in what order, how it recovered, and a wrong final answer and a right-by-luck final answer look identical until you score the path. Here is the rundown as of June 2026.

The final answer is not the unit of evaluation

An LLM-call eval grades one output. An agent eval has to grade a sequence: did it call the right tool, with the right arguments, in a sensible order, and recover when a call failed. Two runs can produce the same final answer, one by reasoning correctly and one by luck, and only trajectory-level scoring tells them apart. If your agent eval only looks at the final response, you are testing a chatbot, not an agent.

The six, by how deep they score

LangSmith. The LangChain-native pick. Agent traces plus eval, automatic if you are on LangChain or LangGraph. Deep on traces, proprietary and coupled to that stack.

Galileo. The agent-focused eval pick. Built around agentic workflows with metrics aimed at tool use and task completion, managed.

Arize Phoenix. The open-source OTel pick. Span-level agent traces plus eval, self-hostable, good if you want trajectory visibility without a license.

Braintrust. The polished-SaaS pick. Strong eval and observability UI for agents, proprietary, no self-host.

Future AGI. The simulate-then-score pick. Their Simulation runs synthetic voice or text personas through your agent before prod, and agentic_eval scores the multi-turn trajectory, tool calls, stepwise reasoning, and the full conversation, not just the final output (github.com/future-agi, as of June 2026). The draw for me was running a synthetic-persona session through the agent like an integration test and then scoring the path it took, not only where it ended up. It is one option among several here, not the answer.

Langfuse. The open-source observability pick. Agent traces plus eval, self-hostable, framework-agnostic; the eval layer is lighter than the eval-specialist tools.

I am not crowning one. LangSmith if you live in LangChain, Phoenix or Langfuse for self-hosted OTel traces, Galileo or Braintrust for managed agent metrics, the simulate-then-score approach if you want to generate the sessions, not just observe them.

What I actually score on a trajectory

Tool-selection-correct (right tool for the step), tool-args-valid, recovery (did it handle a failed call gracefully), and only then final-answer-correct. The first three catch the agent-specific failures the final-answer score hides. The agent that reached a fine answer through three wrong tool calls is a latent incident, not a pass.

Objections I'd accept / wouldn't

Accept: "single-turn metrics still matter." They do. They grade each response, and you want them. They just miss the cross-turn failures (state, tool ordering) that are the whole reason you built an agent rather than a chatbot, so they are necessary and not sufficient.

Wouldn't accept: "trajectory scoring is overkill, ship on final-answer accuracy." That is the position that produces the right-by-luck pass. The agent that stumbles to a correct answer through three wrong tool calls will fail differently next week, and your final-answer metric will not have warned you.

Where I'd push back on this

Steelmanning against myself: trajectory scoring assumes I know what the right path looks like, and for open-ended agents there is often more than one valid path to a good answer. A lot of what I call "wrong trajectory" might be "a reasonable path I did not anticipate," and if I over-fit my eval to one golden path I will punish agents for being creative in ways that are actually fine. The concession: I do not have a clean way to score "took a reasonable path I did not anticipate" without hand-labeling every trajectory. What I hold onto is narrower than full-path matching: tool-args-valid and graceful-recovery are path-independent, they are correct or not regardless of which route the agent took, so I trust those two even when I cannot agree on the one true path. If you have a way to score path-reasonableness without hand-labeling everything, that is the comment I want.

Pydantic passed. Types matched. The downstream system still got garbage.

James O'Connor — Thu, 25 Jun 2026 07:01:37 +0000

I want to walk through three production failures on the same contract-extraction agent, because they looked unrelated at the time and turned out to be the same problem wearing different clothes. My claim, stated up front so you can disagree with it early: schema validation tells you the grammar is correct and nothing about whether the meaning is. Those are two different jobs, and most teams (mine included, for a while) only build the first one.

Case 1: valid JSON, wrong semantics

The extractor used Claude 3.5 Sonnet with Pydantic schemas. A termination_clauses field accepted list[str]. Validation passed every time. The trouble was the model returned paraphrases, not verbatim clause text, and the downstream tool did exact-string matching against a database. Paraphrases never matched.

Pydantic had no way to catch this. The schema said list[str]. Strings arrived. Valid. The fix was a second-pass semantic check (a model call with a rubric asking, in effect, "are these strings verbatim from the source?"). Success on that field moved from 61% to 94%.

Lesson: structured-output validation is syntax validation. Semantic validation is a separate layer (and you have to build it on purpose).

Case 2: the retry cost spike

Retry logic via tenacity. One customer's documents carried a dual-signatory clause with an optional co-signer. The schema expected co_signer: Optional[str]; the model kept returning nested objects instead. Each retry was about $0.04, and on the worst documents that compounded past $2 each before anything escalated.

Two changes: cap retries at 5 with escalation to human review, and audit any new document type before it hits production.

Lesson: unlimited retry logic on validation failures is a latent billing incident (it just hasn't billed you yet).

Case 3: the model-switch regression

We moved GPT-4o to GPT-4.5. Success on party_obligations (a field that needs three-level nesting for conditional logic) fell from 91% to 73%. The newer model handled ambiguous cases with flatter structures. Valid JSON, wrong nesting, Pydantic waved it through, downstream broke quietly.

The fix was shadow evaluation after any upgrade: run old and new models against the same production documents, and flag any field where agreement drops below 95% before shipping.

Lesson: model upgrades are schema-compatibility events (treat them like a dependency bump, not a free swap).

The common thread

None of these surfaced as a Pydantic error. The schema was valid each time. The real failures were semantic drift, an uncontrolled retry loop, and a model-specific regression. In every case the grammar was fine and the meaning was not, which is precisely the thing type validation cannot see.

What the stack looks like now: Pydantic for syntax, a lightweight evaluator for semantics, DeepEval's correctness metric for the text fields, retries capped, an escalation field on every extraction schema so failure modes are a design-time decision, and a shadow-eval checklist of 200 production documents on any model change.

Objections I'd accept / wouldn't

Accept: "stricter schemas would have caught some of this." Partly true. Enums, discriminated unions, and constrained types genuinely shrink the semantic-validation surface when your domain is stable and bounded. If that's you, lean on them.

Wouldn't accept: "so you don't need eval around structured output." Three production failures, two of them customer escalations, disagree. Stricter types reduce the surface; they do not remove it, and they get brittle the moment a new document shape arrives.

Where I'd push back on this

If I'm steelmanning the opposite of my own thesis: maybe the honest read is that I under-specified my schemas and called it a semantics problem to feel better about it. A verbatim-quote field could have been a constrained type backed by a span reference into the source, not a free str. A lot of what I'm calling "semantic validation" is really "validation I was too lazy to encode structurally."

So here's the concession. If you have shipped high-volume extraction without a semantic eval layer and held accuracy above 92% for more than six months, I'd genuinely like to see the schema design, because either you bounded the domain harder than I did, or you encoded meaning into types better than I did. The part I won't give up: somewhere, a field has to assert meaning, and if it isn't your schema doing it, it has to be something downstream of the schema.

I put 6 LLM guardrail tools inline and measured what they cost me. Here is the latency-vs-recall tradeoff.

James O'Connor — Thu, 18 Jun 2026 05:50:48 +0000

An input guardrail runs on every request. Too slow and you rip it out; fast but blind and you get owned. That tradeoff, not the feature list, is the whole decision.

TL;DR: I ran six guardrail and prompt-injection tools inline on a production agent for a few weeks (Lakera Guard, Llama Guard, NeMo Guardrails, Guardrails AI, Future AGI's fi.evals scanners, and ProtectAI's LLM Guard). The deciding axis was not which one detects the most attack types, it was which one was fast enough to run on every request without anyone noticing, while still catching the injections that mattered. Here is the rundown as of June 2026.

An input guardrail sits on the hot path, so latency is the spec

A guardrail that inspects every prompt before it reaches the model adds to every request. Anything over about 50ms inline and users feel it; over about 200ms and someone disables it during an incident. So the real spec is narrow: catch the attack classes you care about (jailbreak, injection, PII or secret leak) inside a latency budget you can afford on the hot path. A 99 percent recall guardrail that adds 400ms is worse in practice than a 95 percent one at 10ms, because the slow one gets turned off.

The six, on the latency-vs-recall axis

Lakera Guard: the commercial-API pick. Strong prompt-injection detection, hosted, low effort to integrate. The tradeoff is a network hop per call (latency plus a third party in your request path) and per-call cost.
Llama Guard: Meta's open LLM-based safeguard model. Flexible policy taxonomy, runs on your own infra. It is an LLM, so it is the heaviest of these on latency unless you serve it carefully.
NeMo Guardrails: NVIDIA's open-source programmable rails (you write flows in Colang). Powerful for conversational and topical boundaries; more of a framework than a drop-in scanner, with the setup cost to match.
Future AGI fi.evals scanners: the inline-speed pick, from their Apache-2.0 ai-evaluation SDK (github.com/future-agi). Local scanners for jailbreak, code injection, PII, and secrets that block in under 10ms and tell you what tripped via result.blocked_by, as of June 2026. The draw was the latency: it runs on the hot path with no network hop, and the managed tier adds model-backed ensemble guardrails on top. Worth saying plainly: these cover attack and safety classes, not business-rule semantic checks.
Guardrails AI: the open-source validation-framework pick. A library of validators (structure, PII, toxicity) you compose; some are fast, some call a model, so your latency depends on which you switch on.
ProtectAI LLM Guard: open-source scanners for input and output (prompt injection, secrets, toxicity). Similar shape to a scanner pipeline; benchmark it against your own latency budget.

I am not crowning one. For lowest-effort hosted detection it was Lakera; for policy flexibility on your own infra, Llama Guard or NeMo; for inline speed with no network hop, the local-scanner approach. They sit at different points on the same curve.

What I gate on, and what I only log

Hard-gate (block the request) on the cheap, high-precision classes: secret and API-key leaks, obvious jailbreak strings, code injection. Log-and-alert (do not block) on the fuzzy classes where a false positive is worse than a miss, because blocking a legitimate user is its own incident. The split is by "how bad is a false positive here," the same logic as eval gating.

FAQ

Inline or async? The cheap deterministic scanners go inline on the hot path; the heavy model-based ones run async or on a sample, unless you can afford the latency.
Do these catch business-logic abuse? No. They catch attack and safety classes (injection, PII, secrets). "The agent did something it should not for THIS user" is a semantic and authorization check you still have to write.
One tool or several? Usually a fast local scanner inline plus a heavier model-based check async. Different tools for different points on the curve.

Open question

Every one of these catches the attack classes you name in advance. The injection that gets through is the one shaped like a class you did not configure, and injection is adversarial, so the attack distribution shifts under you. I do not have a clean way to catch the novel injection that matches no configured scanner. If you have, that is the comment I want.

Bounded retries for agent tool calls: the budget that stopped our infinite-loop incidents

James O'Connor — Mon, 15 Jun 2026 15:11:50 +0000

The worst incident our agent caused was not a wrong answer. It was a loop.

The worst incident our agent ever caused was not a wrong answer. It was a loop. A tool call failed, the agent retried, the retry failed the same way, and it kept going, burning tokens and hammering a downstream API a few hundred times in a minute before anything stopped it. The agent was doing exactly what we had told it: if a tool fails, try again. We just never told it when to stop.

Retries are the right instinct. A transient failure should be retried. The problem is that an agent does not reliably distinguish "transient" from "this will fail every time," and left to its own judgment it will cheerfully retry a permanently broken call until something external kills it. Human-written code learned this lesson decades ago: every retry loop has a bound. Agent tool-calling quietly forgot it, because the retry decision moved from a for-loop you can see into the model's reasoning, where it is invisible and unbounded.

So we put the bound back, outside the model. Two budgets: a per-call retry cap (this specific tool call gets N attempts, then it is a hard failure the agent must handle differently), and a per-session attempt budget (the whole task gets a ceiling on total tool calls, after which we stop and escalate rather than let it spin).

class ToolBudget:
    def __init__(self, per_call=2, per_session=40):
        self.per_call, self.per_session = per_call, per_session
        self.session_calls = 0
    def check(self, tool, attempt):
        self.session_calls += 1
        if attempt > self.per_call:
            raise ToolGivingUp(f"{tool}: {attempt} attempts, stop retrying, try another path")
        if self.session_calls > self.per_session:
            raise SessionGivingUp("tool budget exhausted, escalate to a human")

The important part is not the cap, it is what happens at the cap. A retry that just stops leaves the agent stuck. So when the per-call budget is exhausted we hand the agent a specific message ("this call has failed twice, do not retry it, try a different approach or ask for help"), which turns a dead loop into a decision. Most of the time the agent then does something sensible, because now it knows retrying is off the table.

Tool-misuse loops in our logs went from a handful of nasty incidents a month to basically none, and the few that remain hit the session budget and escalate cleanly instead of paging someone at 2am about a runaway API bill.

The tension I have not fully resolved: a per-session budget that is too tight kills legitimately long tasks (a genuine multi-step workflow can need a lot of tool calls), and one that is too loose lets a slow loop run up real cost before it trips. We set ours from the 95th percentile of healthy task lengths and pad it, which is empirical and a little arbitrary. If you have found a non-arbitrary way to bound agent tool-call budgets, that is the comment I am reading.

We version our tool schemas like an API contract, because the agent is a consumer

James O'Connor — Mon, 15 Jun 2026 05:40:22 +0000

TL;DR: We changed a tool's return schema, shipped it, and watched about 1 in 5 of that tool's calls start failing downstream, even though every call still validated. The schema was internal, so nobody treated the change like a breaking API change. But the agent is a consumer of that schema exactly like an external client is, and we had just shipped a breaking change to a consumer with no version, no deprecation, no migration. Now we version tool schemas like the public contracts they actually are.

The agent is a client you forgot you had

When a human team consumes your API, you version it, you deprecate fields with notice, you do not rename a field on a Tuesday. When the consumer is your own agent, all of that discipline evaporates, because the schema lives in your repo and feels internal. It is not internal. The agent was trained, prompted, or wired against the old shape, and renaming order_id to orderId is as breaking for it as for any client, just quieter, because it fails as worse decisions rather than a 400.

What a breaking change looks like to an agent

This is the part that makes it sneaky. A human client gets a hard error on a missing field. An agent gets a None, treats it as "the value is absent," and makes a plausible wrong decision with a perfectly valid-looking tool call. Our validation passed every time. The damage showed up two hops later as the agent choosing the wrong branch because a field it depended on had silently moved.

How we version them now

@dataclass
class ToolSchemaV2:
    version: str = "2"
    order_id: str          # kept; do NOT rename without a major bump
    status: str
    # added in v2, optional so v1 consumers still parse
    refund_window_closes_at: date | None = None

Rules we adopted, lifted straight from API practice: additive changes are a minor bump and optional; renames and removals are a major bump with both shapes served during a deprecation window; the agent's expected schema version is pinned and checked, so a mismatch is a loud failure at the boundary instead of a quiet wrong decision two hops later.

The open question

Deprecation windows assume you can run two shapes at once, which is fine for fields but hard for semantics: if the meaning of a field changes rather than its name, there is no optional-field trick that saves you. I do not have a clean way to version a semantic change to a tool's contract, only structural ones. If you have versioned a meaning change to an agent's tool cleanly, that is the comment I want to read.

We logged every rejected tool call for a month. A third were our validation being wrong, not the model.

James O'Connor — Mon, 15 Jun 2026 05:33:09 +0000

TL;DR: Everyone logs tool calls that error or return junk. We started logging the calls our own validation REJECTED before they ever ran. Over a month, about 1 in 3 of those rejections were false: a valid user intent our schema or precheck was too rigid to accept. We had spent weeks hardening the guardrail and never checked whether it was now blocking real work.

The blind spot in "we added validation"

After an incident where our agent made a structurally valid but wrong tool call, we added a precheck layer in front of every state-mutating tool. Failures dropped, we moved on. What we did not log was the other side of the ledger: every time the precheck said no. A block felt like a success by definition. The agent tried something bad, we stopped it, good.

Then support started forwarding tickets where the agent refused something the user was clearly allowed to do.

What the rejection log showed

So we logged every rejection with three fields: which check fired, the full arguments, and the user-visible outcome. One month, 612 rejections. We hand-reviewed a sample.

Roughly a third were false rejections. The pattern was almost always the same: a check written to stop one specific bad case was also catching a legitimate neighbouring case nobody thought about when they wrote it. The "is this order in the cancellation window" check rejected legitimate cancellations on orders whose timezone put them one hour outside a window they were actually inside. The "does this id exist in retrieved context" check rejected valid ids that arrived through a second tool the author had not considered.

def run_check(check, args, ctx):
    failures = check(args, ctx)
    log.info("tool_precheck", extra={
        "check": check.__name__,
        "rejected": bool(failures),
        "reasons": failures,
        "args": redact(args),
        "outcome": "blocked" if failures else "passed",
    })
    return failures
# the 'rejected' branch is the one nobody reads. read it weekly.

What we changed

Two things. First, a weekly fifteen-minute review of a sample of rejections, same as we review errors. False rejections get the check loosened or split. Second, checks now fail with a specific reason string the agent can act on, not a generic block, so a too-strict check often self-corrects: the agent reads "outside cancellation window by your local timezone" and escalates instead of dead-ending.

False rejections fell from a third to under a tenth over six weeks. The number that matters more: support tickets about the agent refusing valid requests basically stopped.

The tension I have not resolved

Every loosened check is a check that now lets more through, which is the exact surface the check was added to close. We have not found a principled way to loosen a guardrail without quietly reopening the hole. Right now we lean on the canonical-examples test (the bad case that prompted the check stays frozen as a must-block), but that only protects against the failures we have already seen.

If you run guardrails on an agent: do you measure your false-rejection rate at all, and if so, how do you loosen a check without trusting that a frozen example covers the regression? That is the part I keep getting wrong.

Your schema validation passes and the agent still picks the wrong tool. The bug is semantic.

James O'Connor — Wed, 10 Jun 2026 04:05:27 +0000

Pydantic and JSON-schema guarantee the shape of a tool call. They say nothing about whether it was the right call for the user's intent.

TL;DR: We put strict Pydantic validation on every tool call our agent makes, expecting tool-call failures to drop. They barely did. When I categorized 40 logged failures, 31 of them passed schema validation cleanly. They were well-formed calls to the wrong tool, or the right tool with arguments that were valid types but wrong values. Schema validation catches structural errors. Our actual problem was semantic, and the validator is blind to it.

What schema validation actually guarantees

Pydantic checks types, required fields, enums, ranges. A call like cancel_order(order_id="A123") is structurally perfect even when the user asked to cancel a subscription, not an order. The validator passes it. The user is still angry. Shape is not intent.

The 40-failure breakdown

Of 40 tool-call failures we logged over a few weeks:

9 were real schema violations the validator caught (working as intended).
18 were the wrong tool for the intent, all schema-valid.
13 were the right tool with a semantically wrong argument (valid type, wrong value).

So 31 of 40 sailed through validation. The thing we added to fix tool-call failures addressed less than a quarter of them.

A cheap semantic precheck that helped

After structural validation passes, run a deterministic check that the call's preconditions match the resolved state. No model required.

def precheck(call, ctx):
    # structural validation already passed; now check semantics vs resolved state
    if call.tool == "cancel_order":
        o = ctx.orders.get(call.args.order_id)
        return o is not None and o.user_id == ctx.user_id and o.status in CANCELABLE
    # ... one branch per destructive tool
    return True

This killed the 13 wrong-argument cases almost entirely: the id was valid as a string but did not resolve to a cancelable order owned by this user.

The case this does not solve

The wrong-tool-for-intent bucket (the 18) is harder. Detecting that the agent chose cancel_order when the user meant cancel_subscription is itself an intent-understanding problem, and using another model to judge it just inherits the same blind spot. We stopped trying to verify intent automatically for destructive tools and put a one-line confirmation step in front of them instead.

Open question

How do you test that an agent picked the right tool, not just a well-formed one, without leaning on an LLM judge that shares the failure mode? The precheck handles arguments. Tool selection itself I still gate behind a human-style confirmation, which feels like admitting defeat.

FAQ

Doesn't an LLM judge catch the wrong-tool case? Sometimes, but it misreads intent the same way the agent did, so we do not trust it on the destructive path.
Which model? Genericize: the agent and any judge should be from different model families, but the precheck above is model-agnostic on purpose.

Validate your Pydantic schema before the LLM call, not after.

James O'Connor — Tue, 09 Jun 2026 08:12:31 +0000

A small change that cut our schema-related retries: validate the Pydantic model before sending the request, not after the LLM responds. The usual flow is call, parse, catch the validation error, retry. That burns a full token budget before you learn the schema was wrong. Instead we instantiate the target model with dummy data at boot and on every schema change, and dry-parse a known-good example before the real call. If the schema itself is broken (a bad discriminator, a wrong field type, a renamed enum) it fails in CI or at boot, not on a paid call. Two lines, zero runtime cost, and it caught about 60 percent of our schema bugs before they reached the model. The other 40 percent are genuine model failures, and those are the ones worth retrying. Separating 'my schema is wrong' from 'the model is wrong' is the whole point. Most retry loops conflate them and pay for it in tokens.

MCP tool naming: 6 patterns ranked by how well they survive a refactor

James O'Connor — Fri, 05 Jun 2026 05:29:55 +0000

What happens to your agent when the team renames users to accounts, and why the tool name you picked six months ago decides whether anything breaks

TL;DR: I have shipped MCP servers where the tool names were a thin shell over the underlying REST API, and I have shipped servers where the names came from the domain model instead. The domain-named ones survived backend refactors with close to zero churn. The pass-through ones broke every time someone renamed a table or split a service. After ranking six naming patterns by two axes (how well a name survives a refactor of the system underneath it, and how cleanly the model can pick the right tool), my house pick is ubiquitous-language naming inside a bounded-context prefix, with Pydantic discriminated-union return types doing the schema work. The one-line version of my opinion: Domain-Driven Design plus tool-use schemas is the production fix for agents. The MCP layer is where the anti-corruption-layer belongs, not an afterthought you bolt on later.

A quick scoping note. I am going to use a refund as a naming example throughout, because everyone has a mental model of what a refund is. I am not describing a system that lets a model issue refunds on its own. The refund here is a stand-in for any operation whose name you have to choose. Treat it as a label, not as a production design I am endorsing.

The two axes

Refactor-survival is whether the name stays correct when the system underneath it changes. A name with high refactor-survival describes something stable (an intent in the business domain) rather than something volatile (the current shape of the database or the current REST route).

LLM-selection clarity is whether the model reliably picks the right tool from the name and description alone. Names that are too clever, too abstract, or too collision-prone make the model hesitate or pick wrong. This axis sometimes rewards the literal names that the first axis punishes, which is the whole tension.

#	Pattern	Example	Refactor-survival	LLM-selection clarity
1	Pass-through (mirrors the API)	create_user, delete_user	Low	High
2	Verb-first	process_refund	Medium	High
3	Noun-first / resource-dot-action	refund.create	Medium	Medium
4	Domain-prefix + bounded context	billing.refund.process	High	Medium-High
5	Ubiquitous-language naming	deactivate_account (not delete)	High	High
6	Schema-first (Pydantic discriminated union)	name plus typed return	High	High (with caveats)

The ranking I defend, best to worst on the combined axes: 5, then 6, then 4, then 2, then 3, then 1. Patterns 5 and 6 are not rivals. You use them together.

1. Pass-through naming

The backend has POST /users, PATCH /users/{id}, DELETE /users/{id}, so the tools become create_user, update_user, delete_user. Fast to write, the model reads it fine. Failure mode: the name is welded to the current API surface. The day someone decides a user is really an account, the underlying route changes and your tool name is now a lie. If you rename the tool, every prompt and eval fixture and agent that learned create_user has to be updated. If you do not, new engineers read create_user and go looking for a users table that no longer exists. High on clarity today, low on survival tomorrow.

2. Verb-first

process_refund, cancel_subscription, send_invoice. The action leads, which reads well because tool selection is fundamentally a verb-matching problem. A good verb often describes intent rather than transport, so a backend refactor can leave the name intact. Failure mode: verbs collide and drift as the surface grows. You add process_payment, process_payout, process_chargeback, and now process carries four meanings. process is a weak verb, so people reach for it whenever they cannot think of the precise word. Medium survival, high clarity that decays as the tool count climbs.

3. Noun-first, resource-dot-action

refund.create, subscription.cancel. Groups nicely for a human browsing the list. Failure mode: it optimizes for the resource taxonomy at the cost of the thing the model is best at, verb matching. With the action second, the model reads refund first and has to hold it before the verb that tells it what to do. The grouping also tempts you into CRUD-over-resources when the domain operation you want is richer than create/update/delete. The pattern I reach for least.

4. Domain-prefix with a bounded context

billing.refund.process, identity.account.deactivate. The first segment names the bounded context (the term is from Domain-Driven Design): the part of the business this tool lives in. Namespacing that means something: a billing.transfer and an inventory.transfer stay apart for both model and reader. And a bounded context is a deliberately stable concept, so the prefix survives refactors that wreck pass-through names. Failure mode: the prefix is only as good as your context boundaries, and most teams have not drawn them. If billing and payments and invoicing are three overlapping prefixes nobody can distinguish, you have added ceremony without clarity. The other failure is verbosity. Used with real boundaries it is excellent. Used as decoration it is worse than a flat verb-first name.

5. Ubiquitous-language naming

The one I will defend hardest. Name the tool after the word the domain experts actually use, not the database verb. Do not call it delete_account. In almost every real billing or identity domain you do not delete an account, you deactivate it or close it, and the row sticks around for compliance and audit. So the tool is deactivate_account, and that name encodes a true fact about the domain that delete hides. Two good things happen: the model gets a precise verb (deactivate is far less collision-prone than delete or process), and the name stays correct across refactors because the business meaning does not change when you swap the database. Failure mode: it requires that a ubiquitous language actually exists, and on a lot of teams it does not, or there are three competing dialects. And there is a discipline cost: someone has to resist the easy delete and insist on the accurate deactivate in code review, every time.

6. Schema-first with Pydantic discriminated unions

The first five patterns are about the name. This one is about the return type, and it is where the leverage actually is. A vague return schema (a bare dict, a stringified blob) undoes all the discipline you put into the name. A discriminated union lets one tool return several clearly-distinguished outcomes, each with its own typed shape, tagged by a literal field the model can branch on.

from typing import Literal, Annotated, Union
from decimal import Decimal
from pydantic import BaseModel, Field

class RefundIssued(BaseModel):
status: Literal["issued"] = "issued"
refund_id: str
amount: Decimal
currency: str = Field(min_length=3, max_length=3)

class RefundPending(BaseModel):
status: Literal["pending_review"] = "pending_review"
request_id: str
reason: str

class RefundRejected(BaseModel):
status: Literal["rejected"] = "rejected"
code: Literal["already_refunded", "outside_window", "amount_exceeds_charge"]
message: str

RefundOutcome = Annotated[
Union[RefundIssued, RefundPending, RefundRejected],
Field(discriminator="status"),
]

class RefundResult(BaseModel):
outcome: RefundOutcome

@mcp.tool()

def request_account_refund(charge_id: str, amount: Decimal) -> RefundResult:
"""Request a refund against a charge. Returns one of three outcomes:
issued, pending_review, or rejected (with a reason code)."""
...

The model reads three named outcomes with three shapes, branches on status, and never parses prose to find out what happened. Failure mode: discriminated unions are easy to over-build. Nine variants where three would do is a decision tree the model did not need. The other trap is letting the union drift out of sync with reality, at which point the model gets a typed promise the system cannot keep, which is worse than an honest untyped blob.

House pick

My default, for any MCP server fronting a system I expect to change, is pattern 5 inside a pattern 4 prefix, with pattern 6 on the return side. Concretely: billing.deactivate_account, returning a typed discriminated union. The reason I keep coming back to it is the anti-corruption layer. In DDD an anti-corruption-layer is the translation seam between your clean domain model and a messier external system. An MCP server sits in exactly that position between a model and your backend. If you name tools after the backend, you have no anti-corruption-layer, you have a passthrough, and every backend change corrupts the model's view of your system. That is the whole argument for why Domain-Driven Design plus tool-use schemas is the production fix for agents.

One number, modest and from one project: when we moved a cluster of pass-through tool names over to ubiquitous-language names on a server with roughly thirty tools, our internal tool-selection eval went from about 88 percent to about 94 percent correct-tool-on-first-try. I would not over-read a single before/after on one codebase. The renames that helped most were the ones that killed verb collisions (three different process_* tools) and the ones that replaced delete with the accurate domain verb.

FAQ

Is dotted naming like billing.refund.process even valid for an MCP tool name? Depends on the server framework and client. Some accept dots, others constrain names and you simulate the hierarchy with underscores. Check what your server and client allow. The principle (a stable context prefix) survives whichever character you use.

Will a discriminated union confuse the model more than a flat dict? In my experience it does the opposite, as long as the variants are genuinely distinct and few. The confusion comes from too many variants, not from the union itself.

How is ubiquitous-language naming different from just picking a good verb? A good verb is a writing instinct. Ubiquitous language is a sourcing rule: the verb has to be the one the domain experts actually use, verified against how the business talks, not invented at your desk.

Do I need DDD to get value here? No. You can adopt ubiquitous-language naming and typed returns without ever drawing a context map. The bounded-context prefix specifically only pays off if you have real boundaries.

Open questions I am still chewing on

When a discriminated-union return genuinely has to gain a fourth variant, what is the least disruptive way to roll it out to agents that already learned the three-variant shape? If two bounded contexts legitimately share a verb and a noun, is the prefix enough, or does the duplication signal the boundary is drawn wrong? And the one I go back and forth on: how much naming discipline is worth it before the tool count is high enough to matter. On a five-tool server, pass-through names are fine and contexts and unions are overhead. Somewhere between five and fifty the calculus flips, and I do not have a clean threshold.

Pydantic V2 discriminator pattern for MCP return types

James O'Connor — Tue, 02 Jun 2026 14:55:09 +0000

When an MCP tool can return one of several shapes (success, partial-success, error), the cleanest typing is a Pydantic V2 discriminated union. The pattern survived our last 3 refactors of the tool-return surface.

from typing import Literal, Annotated
from pydantic import BaseModel, Field

class RefundSuccess(BaseModel):
    status: Literal["success"] = "success"
    refund_id: str
    amount_refunded: float

class RefundPartial(BaseModel):
    status: Literal["partial"] = "partial"
    refund_id: str
    amount_refunded: float
    amount_remaining: float
    reason: str

class RefundError(BaseModel):
    status: Literal["error"] = "error"
    error_code: str
    message: str

RefundResult = Annotated[
    RefundSuccess | RefundPartial | RefundError,
    Field(discriminator="status"),
]

Why this beats the alternatives:

The literal discriminator (status) is the field MCP clients (and downstream LLMs) parse first. Pydantic builds the right model based on that field. No if-elif branching in the client.

Subclassing a common parent works but loses the structural specificity. The LLM gets a parent type and has to guess the actual shape.

Optional fields with None for the missing branches works but the type signature lies to readers. The discriminator approach makes the shape obvious.

The gotcha: keep the discriminator field name consistent across the whole MCP server. Pydantic V2's discriminator= is per-union; if one union uses status and another uses type, contributors will mix them up. Pick one and document it in the contributor guide.