Building an MCP tool-call test rig with the Python SDK in 2026

You can test an agent's tool-call loop without a model. Write down the calls the model would have made, replay them against your real MCP server over stdio, and assert on what comes back. It runs offline in about two seconds, costs nothing per run, and catches renamed tools and schema drift before a customer does. The model is the last thing you should be faking.

The Function Call Flow Simulator I link to below is one I built. I tried five existing playgrounds first and every one wanted an API key before it would render a single tool_use block, which is backwards when the whole point is sketching a flow you haven't paid for yet. Mine runs in the browser, free, no signup, nothing leaves the tab. If you know a better one, tell me and I'll link to it instead.

The goal

Here's the picture I wanted on my screen. One command, one JSON file, six lines of output: each tool call my agent would make during a customer refund, run against the same MCP server that handles production traffic, with a failure line and an exit code of 1 the moment something breaks. No API key. Nothing over the network. Under two seconds.

The model is the easy part to fake. Everything around it is where I keep getting hurt. The loop that reads tool_use blocks, dispatches them, feeds results back, and decides when to stop is ordinary code, and it fails in ordinary ways. Someone renamed refund_order to issue_refund on the server and my agent quietly degraded into apologising to people instead of paying them. A required field got added to a schema and half the calls started coming back with isError set, which my loop passed straight back to the model as though it were a normal result.

Testing that against a live model doesn't work the way people hope. Run the same prompt twice and you get different arguments, sometimes a different tool, sometimes a friendly paragraph and no call at all. That's correct behaviour from the model and useless behaviour for a test. So pin the model's output. Write it down as data. Then the only moving part left is your code.

Setup and the auth you don't need

pip install "mcp==1.13.1" is the entire dependency list. I pin the version because the stdio client's environment handling shifted twice inside the 1.x line and I lost most of an afternoon to it.

For replay you need no key whatsoever. That's the point: no model is in the room. You need a key exactly once, on the day you record a real conversation to seed your first transcript, and after that the JSON file is the fixture and CI never sees a credential.

Three environment gotchas worth knowing before you start:

• The server is spawned as a subprocess over stdio, so it inherits the working directory you launched pytest from, not the directory your test file lives in. Relative paths inside your server config will resolve somewhere surprising.
• Anything your server writes to stdout that isn't JSON-RPC corrupts the stream. One stray print() left in a tool handler kills the session with a parse error that names no file and no line.
• StdioServerParameters.env replaces the child environment rather than merging into it. More on that below, because it cost me 41 minutes and my dignity.

Before I hand-write a transcript I sketch the flow in the Function Call Flow Simulator, which lets me lay out the tool_use and tool_result pairs across several turns and copy the JSON straight out. Faster than getting the nesting right by hand at 1am, which is how I used to do it.

The code

This is the whole runner. It reads a transcript, checks each tool still exists on the server, calls it, and stops on the first thing that looks wrong.

# replay.py - drive a real MCP server with a scripted tool-call transcript.
# pip install "mcp==1.13.1"
# usage: python replay.py flows/refund.json
import asyncio, json, sys
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client, get_default_environment

# What the model would have emitted. Hand-written, or recorded from one real run:
# [{"name": "search_orders", "input": {"customer_id": "cus_8812", "status": "open"}},
#  {"name": "refund_order",  "input": {"order_id": "ord_4471", "amount_cents": 9164}}]
TRANSCRIPT = json.load(open(sys.argv[1]))

SERVER = StdioServerParameters(
    command=sys.executable,
    args=["-m", "billing_mcp"],                 # your real server, unmodified
    # gotcha: env REPLACES the child environment, it does not merge. Drop the
    # get_default_environment() spread and the child loses PATH and dies quietly.
    env={**get_default_environment(), "BILLING_MODE": "sandbox"},
)

async def main() -> int:
    async with stdio_client(SERVER) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            declared = {t.name for t in (await session.list_tools()).tools}

            for step, call in enumerate(TRANSCRIPT, 1):
                if call["name"] not in declared:
                    print(f"step {step}: server no longer declares {call['name']}")
                    return 1
                result = await session.call_tool(call["name"], call["input"])
                # gotcha: a failed tool call does NOT raise. isError is a field.
                if result.isError:
                    print(f"step {step}: {call['name']} returned isError")
                    return 1
                text = next((c.text for c in result.content if c.type == "text"), "")
                print(f"step {step}: {call['name']} ok  {text[:80]}")
    return 0

sys.exit(asyncio.run(main()))

The declared set is doing more work than it looks like. list_tools() is the server telling you what it actually exposes right now, so comparing your transcript against it turns "the model will call a tool that vanished" from a production surprise into a failing test. The amount_cents: 9164 in the sample transcript is deliberate. Round numbers hide off-by-one and unit bugs, and I want $91.64 flowing through, not $100.

Scripted replay against the alternatives

I've run all three of these in anger. The numbers in the cost row are mine, from a week in February when our CI ran the agent suite on every push.

Axis	Scripted replay	Live model in CI	Recorded HTTP cassettes
Cost per 1,000 runs	$0	$63.18	$0 after recording
Same result every run	yes	no	yes
Catches renamed or dropped tools	yes	sometimes	no, the cassette hides it
Catches bad model arguments	no	yes	no
Works with no network	yes	no	yes
Time to first green test	~20 min	~5 min	~90 min

Cassettes look like the obvious answer and they're the one I'd steer you away from. They freeze the server's responses too, so the day your MCP server changes its schema the cassette keeps replaying the old world and your suite stays green while production burns. Scripted replay freezes only the model side and lets the server be real, which is the exact split you want.

The live-model column has one row nobody else can fill: whether the model picks sensible arguments. That's a real question. It's just a different suite, on a different schedule, with a budget cap.

What went wrong the first time

I was wrong about isError. I assumed a failing tool call would raise an exception, because that's what every HTTP client I've used does. It doesn't. call_tool returns a CallToolResult with isError=True and content that reads like a normal text block. My runner was green for three days straight while every single refund step came back with "customer cus_8812 not found". Three days of a passing suite that was testing nothing. I only caught it because a teammate asked why the sandbox ledger was empty.

Then the environment thing. I passed env={"BILLING_MODE": "sandbox"} on its own, assuming it merged with the parent process environment the way subprocess.run does with env=None. It doesn't merge. The child got an environment containing exactly one variable, lost PATH, and failed to launch. What I saw was session.initialize() hanging until the 30 second timeout with zero output, because the child's stderr goes nowhere unless you wire it up. I spent 41 minutes convinced my server had a deadlock in its startup handler. The fix is the two-character spread in the code above.

Last Tuesday I added a sixth step to the refund flow and hit the third one: I'd been asserting with result.content[0].text and a substring check. The server had started returning structuredContent alongside a JSON dump in the text block, so my substring "refunded" matched a field name rather than a status value. The assertion passed for entirely the wrong reason. I don't have a clean rule for this yet beyond "parse the JSON, assert on a key", and honestly I'm still annoyed that the loose version survived as long as it did.

None of these three are bugs in the MCP SDK. They're all me assuming a library behaved like a different library I knew better. Worth budgeting an hour for that on any new protocol.

FAQ

Q: Isn't this just testing my own mocks?

A: No, and that's the whole design. The server is your actual MCP server, running as a real subprocess, hitting your real sandbox database. The only mocked thing is the model's choice of tool and arguments, which is the one component you can't assert on deterministically anyway.

Q: How do I get the first transcript?

A: Run the conversation once for real, log every tool_use block your loop receives, and dump the list to JSON. About 15 lines of throwaway code. After that you edit the file by hand to add edge cases the model never happened to produce.

Q: Does this work for HTTP or SSE servers instead of stdio?

A: Yes. Swap stdio_client for streamablehttp_client and pass a URL. Everything inside the ClientSession block stays byte-for-byte identical, which is the nicest property of the client API.

Q: How do I still test that the model picks the right tool?

A: Separate suite, run nightly rather than per-push, with a hard spend cap and assertions loose enough to tolerate variation (did it call any refund-shaped tool, did it stop after four turns). Keep it away from your fast suite.

Written with AI assistance and human review. Try the tool at aidevhub.io/function-call-simulator.