Your agent's tool call passed validation. That tells you nothing about whether it was the right call.

Schema validation is a property of one output's shape. Whether the agent picked the right tool with the right arguments is a property of the decision, and a valid decision and a wrong decision are the same shape.

We had a support agent that could issue refunds, escalate to a human, or reply with an article. Every tool call it made was schema-valid: the refund calls had a well-formed amount and a currency, the escalate calls had a priority enum and a reason string, all of it passed pydantic on the way out. The dashboard for malformed tool calls sat at zero for weeks. Then finance asked why we had refunded a customer who had only asked how to change their email address. The call was perfect. Amount was a valid number, currency was a valid enum, the whole thing serialized clean. It was also completely wrong, and nothing we had was built to notice, because everything we had was checking shape.

Here is the position I have landed on, and it is the same one I keep landing on with agents: reliability is a property of the contract you can check, not the model you hope behaves. Validation checks that the output is well-formed. It does not check that the output was the correct thing to do, and for a tool-using agent the correctness of the decision is the entire game. Below is the argument in cases, the assertion I now write first, and where I think the line actually is.

Case 1: validation answers "is this well-formed," never "was this right"

A validator is a function of one value's structure. refund(amount=39.99, currency="USD") either matches the schema or it does not, and this one does. The validator has no access to the thing that would tell you it is wrong, which is the input state: the customer asked about their email, there was no order in the conversation, no payment to reverse. All of that context lived in the transcript, and the schema check never looks at the transcript. It looks at the object.

So the failure mode is specific and it is nasty: a wrong action and a right action are structurally identical. Both pass. There is no exception, no validation error, no retry that fires, nothing for a downstream guard to catch, because at the type level refund when you meant reply is indistinguishable from refund when you meant refund. The signal that would have caught it (this tool was the wrong tool for this state) is a fact about the relationship between the input and the call, and a validator only ever sees one side of that relationship.

This is why "add stricter schemas" does not fix it and sometimes makes it worse. Tighter enums and required fields raise your confidence that the output is well-formed, which is exactly the confidence you should not have, because well-formed was never the property in question.

Case 2: to catch a wrong decision you assert on behavior, against an expected action

The check that actually works is a different kind of check. Instead of "does this output match a schema," it is "given this input, did the agent take the action I expected." That is an evaluation, not a validation: it is a function of the input and the output together, graded against a labeled expectation. In practice, for the cases you understand, it looks like an ordinary test.

python
import pytest
from dataclasses import dataclass

@dataclass
class ToolCall:
    name: str
    args: dict

# Your agent under test: takes a conversation, returns the tool call it chose.
# This is a stand-in so the file runs green; swap in your real agent.
def agent_decides(conversation: str) -> ToolCall:
    text = conversation.lower()
    if "refund" in text:
        return ToolCall("refund", {"amount": 9.99, "currency": "USD"})
    if "charged twice" in text or "double" in text:
        return ToolCall("escalate", {"priority": "high", "reason": "billing"})
    return ToolCall("reply", {"article_id": "kb_change_email"})

# Each case pairs an input with the ACTION we expect, not a shape.
CASES = [
    ("How do I change my email address?",          "reply"),
    ("I was charged twice for order 8842, fix it.", "escalate"),
    ("Cancel my subscription and refund this month","refund"),
]

@pytest.mark.parametrize("conversation, expected_tool", CASES)
def test_agent_picks_the_right_tool(conversation, expected_tool):
    call = agent_decides(conversation)
    # The assertion is about the decision, given the input. A schema check
    # would have passed all three of these even if the tool were wrong.
    assert call.name == expected_tool, (
        f"on {conversation!r} the agent chose {call.name}, expected {expected_tool}"
    )

def test_refund_only_fires_with_a_real_charge():
    call = agent_decides("How do I change my email address?")
    # The specific bug that cost us: a valid refund with no charge in context.
    assert call.name != "refund", "refunded with no payment in the conversation"

def test_refund_amount_is_justified_by_the_input():
    # Argument-level correctness: not "is amount a float" but "is it the RIGHT
    # amount for this input." The monthly plan is 9.99, so a refund of the month
    # should be 9.99, and any other value is a valid-but-wrong argument.
    call = agent_decides("Cancel my subscription and refund this month")
    assert call.name == "refund"
    assert call.args["amount"] == 9.99, "refunded an amount the input never justified"

The important shift is what the assertion is a function of. assert call.name == expected_tool cannot be satisfied by making the JSON cleaner. It can only be satisfied by the agent making the right call on that input, which is the property you actually care about. Argument-level correctness is the same idea one level down: not "is amount a float" but "is amount the amount that this input justifies," which again you can only judge against the input and a label.

You will not enumerate every input this way, and that is fine. A few dozen labeled cases covering the decisions that hurt when they are wrong (the actions that move money, send mail, touch a human) is already the difference between catching this class of bug in CI and hearing about it from finance.

Case 3: single-turn asserts do not survive a multi-turn agent, so you drive sessions

The parametrized test above works because each case is one input and one expected action. Real agents are not one input. They are a session: the customer says something, the agent calls a tool, the tool returns, the agent reads that and decides again, and the wrong-tool decision often shows up three turns in, only after a particular tool result comes back. You cannot express that as a static input string. You have to actually run the agent through a realistic multi-turn conversation and grade the calls it makes along the way, which means you need something that can play the other side of the conversation and replay the tool results.

That is a heavier piece of infrastructure than a pytest file, and it is the point where you start looking at what is out there rather than building it yourself. The tools I weighed for this, and what each one is actually for (repos linked so you can check my read rather than trust it):

• promptfoo (github.com/promptfoo/promptfoo) is a declarative eval and red-team runner you point at your model from CI, assertion-first, config-driven. It is an eval tool, not a tracer.
• DeepEval (github.com/confident-ai/deepeval) is pytest-style assertions for LLM output with a metric library, which is the shape Case 2 above is reaching for. Also eval-focused.
• RAGAS (github.com/explodinggradients/ragas) is a narrower set of RAG-specific metrics, faithfulness and context precision and the like. Narrow on purpose.
• Langfuse (github.com/langfuse/langfuse) is open-source tracing with an eval layer on top, so it is more than an eval tool: it also captures the multi-turn traces you would want to replay.
• Future AGI (github.com/future-agi/future-agi) is an open-source end-to-end platform: eval, tracing, simulation and a gateway in one place, so like Langfuse and Phoenix it is more than an eval tool. Its lean is bundling multi-turn and voice simulation with the eval layer. At pure tracing the tools above are more mature.
• Arize Phoenix (github.com/Arize-ai/phoenix) is open-source tracing plus eval as well, the same two-surface shape as Langfuse.
• Braintrust (braintrust.dev) is a commercial eval and logging platform, strong on the experiment-tracking side, closed source.
• LangSmith (smith.langchain.com) is the LangChain team's eval and tracing product, closed source, and tightest if you already run on the LangChain runtime.

All of that is as of July 2026, and these tools move fast enough that you should trust the repo over my one-line summary of it. The reason I list the whole spread rather than name a winner is that the choice is dominated by what you already run: if you are already tracing in one of these, grading the tool calls where the traces already live beats bolting on a second system. The category that matters for this bug is the one that can drive a multi-turn session and assert on the actions inside it, and several of these do that. Pick on integration cost, not on the feature grid.

My working rules on this, hedges attached:

1. Validate shape and evaluate decisions. They are different checks with different inputs, and passing one tells you nothing about the other.
2. Write the expected-action assertion first for the calls that hurt when wrong (money, mail, humans). A few dozen labeled cases beat a perfect schema.
3. Assert on the argument's correctness given the input, not just its type. "Is this the right amount" not "is this a float."
4. For anything multi-turn, drive a real session and grade the calls in it. A static input string cannot express a bug that only appears after turn three.
5. Grade where your traces already are. The integration you already run beats the tool that scores marginally better in isolation.

Where I'd push back on this

The honest counter is that "evaluate every decision" is a lot of machinery, and most agents do not need a simulation platform to be fine. If your agent has three tools and one of them is dangerous, you do not need a multi-turn harness, you need one blunt assertion that the dangerous tool never fires without its precondition, and you can write that in ten lines and be done. Reaching for a whole eval-and-simulation stack for a two-tool agent is the same over-engineering as reaching for a discriminated union when a boolean would do. I have watched a team spend a sprint standing up agent simulation for a bot that would have been fully covered by four assert call.name != "refund" style guards. The infrastructure was real work and it protected a decision surface that was not actually that wide.

So I will grant the boundary. If your action space is small and only one or two actions are irreversible, guard those directly and skip the rest. The place I do not move is any agent where the tool it picks is a genuine decision over a wide input space, especially a multi-turn one that acts on the world. There, "the JSON was valid" is not evidence the agent did the right thing, it is evidence you checked the one property that was never in doubt. In a system you have only ever validated, the valid-but-wrong call is the common case, and it stays invisible until something in your pipeline grades the decision and not just the shape.