DEV Community: Nicola Mastromarino

Why JSON tool calling is breaking your agents (and how MCP Code Mode fixes it)

Nicola Mastromarino — Mon, 09 Nov 2026 11:06:23 +0000

Standard tool calling looks great when your agent has three tools. You pass the JSON schemas in the system prompt, the model emits a tool_call payload, your backend executes it, and you feed the result back.

Then you connect your agent to real enterprise infrastructure.

Suddenly you have 4 MCP servers, 150+ endpoints, and complex multi-step workflows. Before the user even types a prompt, you are burning 80,000 tokens just to load tool definitions into the context window. And when the agent actually needs to do something nontrivial, it gets stuck in a slow, expensive ping-pong loop of sequential JSON calls.

Enter Code Mode—a pattern popularized across Cloudflare Agents, FastMCP, and the Vercel AI SDK that replaces chatty JSON tool-calling chains with single-shot, sandboxed code execution.

Here is why standard JSON tool calling hits a wall in production, and how switching to Code Mode changes the math on latency, tokens, and reliability.

1. The Upfront Context Tax (1M tokens vs. 1,000 tokens)

In standard Model Context Protocol (MCP) or native function calling, the model needs to see the schema of every tool upfront.

To give a concrete benchmark: exposing the full Cloudflare API natively via MCP (around 2,500+ endpoints) consumes roughly 1.17 million tokens of schema definitions. Even stripping it down to required parameters still burns ~244,000 tokens.

Beyond the sheer cost, attention degrades as context grows. When an LLM has to choose between 200 similarly named tools, hallucinated parameters and wrong tool selections spike.

How Code Mode fixes it: Instead of dumping every schema into the prompt, you expose a typed SDK (TypeScript or Python) inside a sandbox and give the model just two meta-tools: search_spec and execute_code (around 1,000 tokens total). The agent discovers only the exact types it needs on the fly and writes code against them.

2. Death by 10 Sequential Round-Trips

Suppose a user asks: "Find all failed deployments from the last 24 hours, check their error logs, and open a ticket for any 5xx database timeout."

With standard JSON tool calling, the execution looks like this:

Model calls list_deployments(status="failed") -> waits for network + inference.
Backend dumps 50 deployment objects into the context window.
Model calls get_logs(deployment_id) 10 separate times -> 10 round-trips, 10x token re-ingestion of the entire conversation history.
Model parses the text in-context and calls create_ticket().

You are paying for full-context prefill on every single hop, and the user is staring at a spinner for 45 seconds.

How Code Mode fixes it: The LLM writes a single script that loops, filters, and aggregates inside the execution environment, returning only the final outcome to the context window:

# Instead of 12 JSON round-trips, the LLM generates and runs this in a sandbox:
async def triage_failed_deployments(sdk):
    deployments = await sdk.deployments.list(status="failed", hours=24)
    created_tickets = []

    for dep in deployments:
        logs = await sdk.logs.get(dep.id)
        if "504 Gateway Timeout" in logs or "DBConnectionError" in logs:
            ticket = await sdk.tickets.create(
                title=f"DB Timeout on deployment {dep.id}",
                priority="high",
                snippet=logs[-500:]
            )
            created_tickets.append(ticket.id)

    return {"triaged": len(deployments), "tickets_opened": created_tickets}

One LLM generation. Zero intermediate JSON payloads polluting the context window.

3. LLMs are better at writing code than chaining JSON

Frontier models have been trained on billions of lines of Python and TypeScript—not on bespoke, deeply nested JSON-RPC tool-calling trajectories.

When you force an agent to orchestrate logic through the conversation history, you are asking the attention mechanism to act as a CPU and RAM simultaneously. When you let the agent write code, you offload control flow (loops, conditionals, retries, parallel calls) to a deterministic runtime.

If you use Python with FastMCP, enabling this pattern over an existing server takes three lines:

from fastmcp import FastMCP
from fastmcp.experimental.transforms.code_mode import CodeMode

# Wraps registered tools into a programmatic sandbox interface
mcp = FastMCP("EnterpriseOps", transforms=[CodeMode()])

4. Where Code Mode breaks (The Red Team view)

Code Mode isn't a silver bullet, and treating it as one introduces a different class of production failures:

Sandbox Security is non-negotiable: You are executing LLM-generated code. exec() on your host container is an incident waiting to happen. You need strict isolation: V8 isolates (Cloudflare Workers), QuickJS WebAssembly sandboxes, MicroVMs (Firecracker / E2B), or gVisor-backed containers with zero ambient network access.
Opaque Mid-Script Failures: If step 4 of a 6-step generated script throws an unhandled exception, you lose the granular step-by-step state visibility you get from standard agent graphs (like LangGraph). Your sandbox must return structured tracebacks and partial execution state so the agent can self-heal without re-running non-idempotent mutations (like double-charging a payment API).
Overkill for Simple Routing: If your agent only has 4 tools and performs single-hop lookups (like a standard RAG retriever), spinning up a sandbox and compiling TypeScript adds unnecessary overhead.

The Architecture Rule of Thumb

Use Direct Tool Calling when:

You have < 15 tools with small, flat schemas.
Calls are single-hop or require human-in-the-loop approval between every individual step.

Use Code Mode when:

You are exposing large APIs or multiple MCP servers where schema tokens exceed 10k+.
Workflows involve loops, data filtering, joins across endpoints, or parallel requests.
You want to keep large intermediate API payloads out of the LLM's context window.

Are you still relying on native JSON tool calling for multi-step agents, or have you started moving orchestration into sandboxed code execution? Let me know what sandboxing stack you're running in the comments.

5 RAG mistakes that looked fine in the demo and broke in production

Nicola Mastromarino — Sat, 03 Oct 2026 05:06:32 +0000

Every RAG demo works. You pick five questions, the system answers them beautifully, and everyone nods.

I've built RAG pipelines over clinical notes and over a knowledge graph, and the gap between "works in the demo" and "works on real questions" is bigger than you'd expect. Real users ask messy questions, use acronyms nobody documented, and expect answers from documents that changed last Tuesday.

Here are five mistakes that pass the demo and fail on real data.

1. Judging quality by vibes

The demo set is five questions you wrote, about documents you know. Of course it works.

Without an evaluation set you can't tell whether a change to chunking or embeddings made things better or just different. On a clinical QA project (about 40K MIMIC-III notes) I tracked Token F1 and hallucination rate against a baseline. That's the only reason I can say QLoRA fine-tuning cut hallucinations by roughly 40% and lifted Token F1 by 190%, instead of "the answers felt better."

The same applies to retrieval. Build a golden set of 30-50 real questions, each mapped to the document that should answer it, and measure retrieval separately from generation. If the right chunk never reaches the model, no prompt will save you.

def hit_rate_at_k(eval_set, retriever, k=5):
    hits = 0
    for item in eval_set:
        docs = retriever(item["question"], k=k)
        if any(d.id in item["relevant_ids"] for d in docs):
            hits += 1
    return hits / len(eval_set)

Twenty lines of code, and it changes how you work.

2. Fixed-size chunking

Splitting every document into 500-token blocks is fast and wrong. It cuts tables in half, separates a list from its heading, and ends chunks mid-sentence. The chunk that says "the limit is 10,000" no longer says what limit.

Chunk along the document's structure (sections, headings, paragraphs), keep the section title attached as metadata, and consider retrieving small chunks but passing the larger parent section to the model.

3. Vector search only

Embeddings are great at meaning and mediocre at exact strings. Error codes, product IDs, contract numbers, internal acronyms: this is exactly what users search for, and exactly where pure semantic search is weakest.

The retriever in my clinical system combines BM25 with a biomedical embedding model, fused with Reciprocal Rank Fusion. Medical text is full of abbreviations and drug names, exactly the kind of string where keyword matching earns its place next to embeddings. Add a reranker on top and you get one of the cheapest upgrades with one of the biggest effects.

4. Letting the model answer no matter what

When retrieval returns nothing useful, most systems still pass the top-k chunks to the LLM, and the LLM still produces a fluent answer. That's the worst failure mode, because it looks like success.

Set a relevance threshold and let the system say "I couldn't find this in the documents."
Require citations, so a wrong answer is at least checkable.
Don't stuff the context with 20 chunks "just in case." More context often means more noise, not more accuracy.

My clinical QA system keeps a human in the loop (it was designed with the EU AI Act in mind). When the domain is medical, a fluent wrong answer isn't a UX bug, it's a risk.

5. Treating the index as finished

The demo index was built once. Real documents change, get deleted, get replaced by newer versions, and some of them shouldn't be visible to every user.

You need an ingestion pipeline, not a script: updates and deletions handled, versions tracked, metadata filters for permissions, and a plan for re-embedding when you change the embedding model. An answer based on a policy that was withdrawn two months ago is worse than no answer.

The pattern behind all five

Every one of these is invisible in a demo and obvious on real data. The demo tests whether the pipeline runs. Real usage tests whether it retrieves the right thing, says no when it should, and stays correct over time.

Build the evaluation set first. Everything else gets easier once you can measure it.

What's the worst RAG failure you've seen that passed the demo? I'd love to hear the war stories in the comments.

I build LLMs for a living. Here's why I wrote a book about slowing down.

Nicola Mastromarino — Mon, 03 Aug 2026 13:30:00 +0000

My job is to make machines think faster.

I work daily with large language models and generative AI systems. I fine-tune, optimize, reduce latency, eliminate friction. My entire professional value is measured in throughput, tokens per second, inference speed.

And then one evening I noticed something: it was 11 PM. I had been "productive" for 14 hours straight. My thumb was scrolling through my phone — screen off. Just muscle memory. My brain was too fried to even choose what to look at, but it kept reaching for stimulation anyway.

That was the moment I realized something was deeply wrong. Not just with me — with how we, as people who build fast systems, have started treating ourselves like fast systems.

The paradox nobody talks about in tech

We spend our careers removing friction, automating repetitive tasks, optimizing pipelines. The promise is always the same: "this will save you time."

But somehow, the more time-saving tools we build and adopt, the less time we feel we have. We're busier than ever. More anxious than ever. More connected than ever, and more exhausted than ever.

I started digging into the neuroscience behind this. What I found changed how I think about work, attention, and time.

The ancient Greeks had two completely different words for time:

Kronos — chronological, measurable, ticking time. The kind you track in Jira tickets and calendar blocks.
Kairos — qualitative, dense, meaningful time. The kind where an hour feels like a minute because you're fully present.

Modern knowledge work, and especially tech culture, has become obsessed with Kronos. We fill every second. We context-switch 50 times a day. We treat rest as wasted clock cycles.

But here's what the neuroscience actually says: your brain does its most important work — creativity, memory consolidation, emotional regulation — precisely when you stop optimizing it.

The Default Mode Network (the part of your brain that activates during rest and "mind wandering") is not idle. It's processing. It's connecting dots you couldn't connect while you were sprinting.

We've been so busy optimizing Kronos that we've completely destroyed our Kairos.

What I did about it

I started writing down protocols for myself. Not productivity hacks. Not "wake up at 5 AM" routines. Just practical, neuroscience-backed ways to protect my attention and reclaim some sense of pace in a job that constantly demands more speed.

Things like:

Monotasking blocks: single-context work sessions where notifications don't exist
Cognitive cooldowns: intentional transition rituals between deep work and shallow work
Digital friction by design: making the default action the slow one, not the fast one

The notes grew. I kept refining them. Eventually they became a short book: The Art of Slowing Down in the Age of Hyper-Speed.

Writing it felt deeply ironic — an AI engineer writing a manifesto against the very culture his work accelerates. But that's exactly the point. We're the ones building the machines. We're the ones who should understand, better than anyone, why the human brain can't run like one.

The one thing I'd leave you with

If you work in AI, ML, or any fast-moving area of tech, you probably already feel this tension. The industry screams FASTER every single day. New model, new framework, new benchmark every week.

You don't have to opt out of all of it. But you do have to protect some space for Kairos. Otherwise, you're just a very expensive server running at 99% CPU with no garbage collection.

And we all know what happens to systems with no garbage collection.

Thanks for reading. If this resonated, I'd love to hear how you deal with the pressure to optimize everything — including yourself.