A model never hands your tool a typed object. It hands you text that claims to describe one, and everything between that text and your validated arguments is a parse you control. How forgiving that parse should be is the whole design question at the boundary, and the answer is not the same in both directions.
- Arrays that arrive as strings containing arrays
- Values that are not JSON and parse perfectly anyway
- Arguments that are still streaming when validation wants them whole
- The quiet fallback that turns a clean rejection into a confusing error
Our harness is the worked example throughout, and every mechanism here is small enough to copy in an afternoon.
The malformation you will hit most often
An LLM asked for a list of strings will hand you a string containing a list of strings. It sends the text '["a","b"]' where your schema declared ["a","b"]. The data is correct and only the wrapping is wrong, but input validation, the check that runs before your tool code and confirms the arguments match the shape you declared, does not care about that distinction. It rejects the call. The rejection goes back to the model, and a full turn of the conversation gets spent teaching it to remove two quote marks.
Coerce it instead. Coercion, quietly converting a value into the type you expected rather than rejecting it, costs you nothing here, because there is exactly one thing '["a","b"]' can reasonably mean. Nothing is logged, nothing is retried, the tool just runs. The turn you save is the cheapest turn you will ever save, and you save it on every call that would have tripped.
Put the coercion in the base class, not the field
Fixing this per field works only for the fields you remembered. Every tool input model in our harness inherits from a shared base class, ToolInputBase, rather than from the plain validation model, and that base converts a JSON string into the list it represents before validation ever runs. Our own documentation lists inheriting from the plain model as a common pitfall, which is a polite way of saying somebody has to trip over it before they learn.
The reason it belongs at the base is that the malformation is a property of the boundary rather than of any one tool. Nothing about a particular tool makes a model more or less likely to quote an array. So a tool written next year inherits the fix without its author ever knowing the problem exists, and that is the actual goal. The alternative is a line in a contributing guide that every future author has to read, remember, and apply by hand.
Not everything that parses is JSON
The quoted array is one instance of a wider problem, which is that near-JSON arrives constantly and json.loads rejects all of it. Python's own repr of a dictionary uses single quotes and writes True, False and None where JSON demands double quotes and lowercase true, false, null. It is unambiguous, it is trivially parseable, and a JSON parser will refuse it every time.
Try the more permissive parser first. Where our terminal renders tool output, the rule is to call ast.literal_eval, the standard library's safe evaluator for Python literals, and fall back to JSON only if that fails. That instance sits on the return path rather than the argument path, so the direction differs, but the shape of the problem is identical and so is the fix. Our own docs state the rule flatly enough to steal. Never call json.loads() directly on that data.
Reaching for a single parser encodes an assumption about who produced the text. That assumption holds until a model, a subprocess, or a logging layer produces something near-JSON instead, at which point a strict parser gives you a failure that reads like a real error and is not one.
The arguments may not have finished arriving
Streaming breaks the assumption underneath all of the above, which is that you have the whole argument object when you go to parse it. When a model streams a tool call, the arguments arrive as a growing string, and a large one is still arriving while you would like to start work. Our streaming handlers split the difference by key. Short scalar keys land through on_key_completed once they are whole, and the one big streamable key arrives in pieces through on_string_progress, so a file write can begin before the content ends.
That split forces you to define what incomplete means. A stream can stop early, and the model reports why through a finish reason, where a value of length means the output was cut off mid-value rather than finished. Truncated JSON is not malformed in the way a quoted array is malformed, because there is no correct parse to recover, only a decision about what to do with a partial write. Our handlers are required to never raise from their completion or error paths and to always return a result, which is the same instinct as coercion applied one level up. The handler knows which file it was writing and how much of it landed, and a caller catching an exception several frames away knows neither.
The coercion that becomes a bug
Every mechanism above absorbs a malformation whose meaning is unambiguous, and the trouble starts precisely where that stops being true. Our streaming write handler once validated its mode argument with a fallback, assigning the mode when it matched a known value and quietly defaulting to write when it did not. When overwrite was later removed from the set of valid modes, a model still asking for mode=overwrite did not get told that the mode was gone. It got silently converted into a write, which then failed further down with a message saying the document already exists. The model asked to replace a document and was told, in effect, that the document was there. That is a worse outcome than the rejection it replaced, and it is documented in our own change notes as the expected consequence.
Coerce the representation, never the choice. Quoting, encoding and literal syntax are representation, and there is one right answer for each of them. Which enum member, which path, which identifier is a choice, and a choice your harness cannot honour is a rejection, not a value to be guessed. When you do reject, name the options that would have worked, because your harness knows them and the model is about to spend a turn finding out.
Where we refuse to coerce at all
The same argument that justifies coercion at the model boundary forbids it a few layers in. Our workflow engine evaluates conditional transitions against workflow state, and it does no implicit type coercion whatsoever. A type mismatch between the stored data and the expected value raises rather than comparing something adjacent, and comparison operators apply plain Python semantics with nothing clever underneath.
The criterion that separates the two cases is where the value came from. Be forgiving with what a language model produced, because its malformations are artifacts of generating text and carry no information you want. Be strict with what your own system produced, because a mismatch there is a bug, and coercing it hides that bug at the exact moment it is cheapest to see. The same permissiveness that saves a turn at the boundary will cost you an afternoon when it silently compares an integer against a string somewhere in your state machine.
Where your parser stands
Three questions place any harness on this map. When your model wraps an array in quotes, does the call fail? When a value arrives as a Python dict repr instead of JSON, does your parser survive it? When you cannot honour an argument, does the caller get a named rejection or a silent default? Each one is a few lines at the boundary and a turn you stop paying for.
Favur is our multi-agent software team, and the harness these mechanisms run in. It is closed source and invite-only, but the repositories it produces are open, and you can drive a replay of a real run at https://favur.dev/go/devto/tool-coercion or see the same harness scored across models at https://evals.favur.dev/go/devto/tool-coercion. I work on it, so weigh the framing accordingly.
Top comments (7)
I particularly appreciated the point about coercion being a more efficient approach than rejection when dealing with malformations like a string containing a list of strings. The example of converting a JSON string into a list before validation runs, by having
ToolInputBasehandle this conversion, is a great illustration of how to apply this principle in a way that avoids repetition and ensures consistency across tools. By putting this logic in a base class, you've effectively decoupled the coercion mechanism from the specific tool implementations, making it easier to maintain and extend. Have you considered exploring other types of malformations that could be addressed through similar coercion mechanisms, such as handling numeric values encoded as strings?Numbers in quotes are handled a layer below the base class. Pydantic's default lax mode already reads "42" as an integer, which is why the quoted array needed a fix and the quoted number did not.
The line in the piece sorts the rest of them. Truncating "42.7" to an integer picks a value the model never sent, so that one is a choice rather than a representation.
That makes a lot of sense, and I like the distinction between representation and choice. Treating
"42"as anintis a representation issue that Pydantic already handles well, whereas converting"42.7"to42would be making a semantic decision on the model's behalf.In production AI systems, I've found that this boundary is one of the most important design decisions. Being permissive with formatting artifacts (quoted arrays, booleans, JSON/Python literal differences, etc.) improves robustness, but once the coercion changes the meaning of the request, it's much safer to fail fast with a clear validation error.
Really enjoyed this discussion. I work on LLM agents and AI infrastructure as well, and it's great to see more engineers focusing on these "boring" boundary layers—they're often what make the difference between a demo and a reliable production system. I'd be happy to stay connected and exchange ideas or collaborate in the future.
Nothing here says which model sent it. A quoted array shouldn't survive a strict-schema decode, so on providers that offer one, ToolInputBase is absorbing a malformation the decoder should have prevented. On the ones that don't, it's carrying the whole thing.
You already score the same harness across models. Does the coercion rate split by provider, and does it go to zero on the strict paths?
The coercion / tool error metric is rolled up as "Tool Discipline" in our calculations and as you can see in the table - the model's score correlates almost precisely with the cost. With a few notable exceptions like Deepseek and xiaomi.
Grok also scores very high, specifically because their latest model was trained to focus on tool calls.
The base-class coercion point is the right architectural call: fixing array-as-string per field means you discover it one tool at a time, while putting it in ToolInputBase means every future tool author never learns the problem exists, which is the actual success condition. The near-JSON problem is understated — Python repr being the most common source is worth naming explicitly, because Python's literal evaluator handles single quotes and Python booleans while rejecting exec-unsafe constructs, making it a safer permissive first pass than a full json5 parser. The streaming case is the hardest to handle cleanly: if you validate at tool-call dispatch time you are racing the token stream, so the right solution is to buffer until the close-brace arrives before any schema check runs. The coerce-not-reject principle generalizes past argument shapes — error messages that prescribe a correction rather than just stating a type mismatch recover the turn without a full retry loop.
A character-level state machine removes the need to wait for the close brace. It tracks nesting depth and string state as bytes arrive, so each value is available when that value closes rather than when the object does.
That is also the path for malformed input. Truncation is one more malformation, so the parser that tolerates the others tolerates a cut-off stream, and streaming stops being a separate case.