I thought I needed a better tool-calling model, but my agent just had too many tools

A few months ago, I would have blamed the model.

Agent picks the wrong function? Fine, upgrade to GPT-5. Try Claude Opus. Add a smarter router. Maybe benchmark Qwen or Llama if you’re running local.

That instinct is everywhere because model swaps feel clean. Refactoring your agent’s tool surface feels like admitting the architecture is a mess.

Then I ran into a couple of OpenClaw threads that made the problem look a lot less like “model intelligence” and a lot more like “bad menu design.”

The bug didn’t look like a model bug

In one r/openclaw thread about hidden tools, a user was trying to get an agent to send a specific image file.

Instead of using the media-sending path, the agent kept reaching for sessions_send, which appeared to be intended for internal agent-to-agent communication.

That’s already bad.

But the more interesting detail was this: the user said the message capability only triggered reliably with a specific prompt, and even then only around 8/10 times. They described it like this:

The tool is hidden in a way where the agent will flat out refuse its existence completely.

That is not a “buy a smarter model” problem.

That is a visibility problem.

Your agent may be doing exactly what you showed it

When people ask for the best model for tool calling, they usually mean:

• Which model is best at choosing the next action?
• Which model recovers best from ambiguous instructions?
• Which model handles a large tool inventory without getting confused?

Those are valid questions.

A stronger model absolutely helps. GPT-5 will usually route better than weaker models. Claude Opus often does better when tool descriptions are fuzzy. Smaller open models can fall apart faster when context gets noisy.

But there’s a more boring explanation for a lot of agent failures:

you exposed the wrong tool, at the wrong time, with the wrong description.

If sessions_send is visible during a normal user request, and message is semi-hidden, the model is not failing an IQ test.

It’s picking from a bad action menu.

This is an architecture problem, not just a model problem

A lot of teams still build agents like this:

• attach every function
• attach every skill
• expose giant schemas
• hope GPT-5 or Claude sorts it out

That works for demos.

It breaks in production.

Once the tool surface gets large, wrong-tool selection starts looking like “the model is dumb” when it’s really:

• too many visible options
• overlapping descriptions
• internal-only tools exposed to user-facing flows
• no task-level routing
• no tracing

If you’ve ever watched an n8n agent, Make scenario, Zapier AI step, or OpenClaw workflow call the weirdest possible function, this is probably familiar.

OpenAI’s guidance quietly points in the same direction

One thing that surprised me: OpenAI’s current function-calling guidance leans toward reducing what the model sees, not just upgrading the model.

If you have lots of functions or large schemas, the recommendation is to avoid making the model evaluate the entire function inventory every turn.

That leads to two practical patterns:

1. Narrow the visible toolset yourself based on task or workflow stage.
2. Use deferred tool loading so rarely used functions are only considered when needed.

That is the opposite of the usual “just expose everything” instinct.

And honestly, it makes sense.

If the user is clearly trying to send a file, why is the model staring at 40 unrelated actions?

The easiest way to break tool calling

Here’s a simple bad example.

TOOLS = [
    send_message,
    sessions_send,
    book_calendar,
    summarize_analytics,
    extract_webpage,
    create_invoice,
    search_docs,
    upload_file,
    transcode_media,
]

response = client.responses.create(
    model="gpt-5.4",
    input="Send the latest screenshot to the customer",
    tools=TOOLS,
)

This is easy to build.

It is also how you end up debugging nonsense at 2 a.m.

Now compare that to task-scoped exposure:

def tools_for_task(task_type: str):
    if task_type == "send_file":
        return [find_file, prepare_attachment, send_message]
    if task_type == "support_lookup":
        return [search_docs, summarize_docs]
    if task_type == "calendar":
        return [check_availability, book_calendar]
    return [fallback_answer]

visible_tools = tools_for_task("send_file")

response = client.responses.create(
    model="gpt-5.4",
    input="Send the latest screenshot to the customer",
    tools=visible_tools,
)

Same model.

Much better odds.

The LangChain mental model is actually useful here

LangChain describes an agent as something like:

model + harness

That harness matters more than people admit.

Your harness includes:

• the system prompt
• the tool list
• tool descriptions
• middleware
• routing logic
• memory setup
• retries and guards

If the harness is sloppy, a stronger model can help, but it won’t fully save you.

The tiny LangChain examples work because the action space is tiny:

from langchain.agents import create_agent

def get_weather(city: str) -> str:
    """Get weather for a given city."""
    return f"It's always sunny in {city}!"

agent = create_agent(
    model="openai:gpt-5.5",
    tools=[get_weather],
    system_prompt="You are a helpful assistant",
)

One tool is easy.

Fifty tools with overlapping semantics is where things get ugly.

Weird output is often a tool-surface bug in disguise

A second OpenClaw thread made this even clearer.

A user reported a bizarre block of corrupted-looking output. Another user replied that Graphify was the culprit:

It would send huge blocks of numbers, random things about Belarus, text in mandarin, and so on. Removed it, problem gone.

That’s a perfect debugging story because it kills the usual fantasy that every weird agent behavior is frontier-model instability.

Sometimes the fix is not:

• lower temperature
• switch models
• rewrite prompts for three days

Sometimes the fix is:

• remove the bad tool
• hide the internal tool
• stop exposing experimental skills in production

That should make you more paranoid about your tool surface.

What actually works better in practice

Approach	What happens in practice
Expose all tools to one agent	Fast to ship, but wrong-tool selection gets much more likely and debugging becomes guesswork
Task-scoped tool exposure	Better reliability because only relevant actions are visible, but you need routing logic
Deferred tool loading	Great when you have a large function inventory, but requires support in your stack and a bit more design work

For broad research agents, wide exposure can make sense.

For most production automations, it usually doesn’t.

That includes:

• n8n workflows
• Make scenarios
• Zapier AI actions
• OpenClaw skills
• support agents
• file-processing workers
• internal ops bots

In those systems, exposing everything is usually laziness disguised as flexibility.

Five things I’d fix before changing models

Before you benchmark GPT-5 vs Claude Opus vs Qwen vs Llama, do this first.

1. Hide internal-only functions completely

If a function is not meant for user-facing requests, don’t leave it in the visible inventory.

Bad:

tools=[send_message, sessions_send, search_docs]

Better:

user_tools=[send_message, search_docs]
internal_tools=[sessions_send]

Put internal tools behind:

• a supervisor
• middleware
• a separate worker
• a sub-agent with its own policy

If the user-facing model can see internal transport primitives, you’re inviting failure.

2. Expose tools by task, not by global capability

A file-send request should not expose unrelated functions just because they exist somewhere in the system.

Good rule:

• file task -> file lookup, attachment prep, send message
• calendar task -> availability, scheduling, confirmation
• research task -> search, fetch, summarize

Don’t make the model sort through your entire company’s API surface every turn.

3. Rewrite tool descriptions like UI labels

Most tool descriptions are terrible.

They read like internal notes from a sleep-deprived engineer.

Bad:

def sessions_send(...):
    """Sends session data."""

Better:

def sessions_send(...):
    """Internal-only transport for agent-to-agent communication. Never use for replying to end users or sending media/files."""

Bad:

def message(...):
    """Message capability."""

Better:

def message(...):
    """Send a user-facing message or media attachment to the external recipient. Use this for normal outbound communication."""

Descriptions are part of the interface.

Treat them that way.

4. Turn on tracing before you start guessing

If you’re using LangChain, enable LangSmith tracing:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=your_key_here

Then inspect:

• which tools were visible
• what the model actually saw
• which tool it selected
• whether the descriptions overlapped
• whether the prompt accidentally encouraged the wrong path

Without tracing, you’re doing folklore engineering.

5. Only then test model upgrades

After the tool surface is clean, then benchmark models.

That’s when the question becomes real:

• Does GPT-5 route better on long-context tasks?
• Does Claude Opus recover better from ambiguity?
• Is Qwen or Llama good enough for this narrower action space?

Before cleanup, those benchmarks are noisy because you’re measuring architecture mistakes as if they were model quality.

A practical routing pattern for automations

If you’re building automations in n8n, Make, Zapier, or a custom worker setup, a simple pattern works well:

1. classify the task
2. expose only the relevant tool subset
3. call the model
4. execute the selected tool
5. trace everything

Example:

def classify_task(user_input: str) -> str:
    if "send" in user_input and "file" in user_input:
        return "send_file"
    if "schedule" in user_input or "calendar" in user_input:
        return "calendar"
    return "general"

TASK_TOOLS = {
    "send_file": [find_file, prepare_attachment, send_message],
    "calendar": [check_availability, book_calendar],
    "general": [search_docs, fallback_answer],
}

task = classify_task("Send the latest screenshot to the customer")
visible_tools = TASK_TOOLS[task]

This is not fancy.

That’s the point.

Boring routing beats magical thinking.

Where Standard Compute fits into this

Once you start cleaning up tool exposure, another issue shows up fast: testing agents properly burns a lot of model calls.

You don’t just run one prompt.

You run:

• repeated tool-selection tests
• routing experiments
• long workflow simulations
• retries across different models
• regression checks after every harness change

That gets expensive fast if you’re paying per token.

This is exactly why I think flat-rate AI infrastructure is underrated for agent teams.

With Standard Compute, you can use an OpenAI-compatible API and run those iterations without watching token spend every hour. It’s especially useful if you’re building agents and automations that need constant testing across GPT-5.4, Claude Opus 4.6, and Grok 4.20.

If your workflow already uses an OpenAI-compatible SDK, the drop-in swap is straightforward.

That matters because tool-routing bugs are rarely fixed in one shot. You need room to test, trace, and rerun.

Per-token pricing makes people under-test the exact systems that most need repeated evaluation.

What if the model really is the problem?

Sometimes it is.

Long context can hurt tool selection. Weak memory setup can pollute intent. Smaller models can struggle with subtle distinctions even when the tool surface is decent.

And yes, stronger models often help a lot.

But I think too many teams ask:

what’s the best model for tool calling?

before they ask:

why can this agent see that function at all?

That second question is less exciting.

It also fixes more bugs.

The takeaway

If your agent keeps choosing the wrong function, don’t start with a model swap.

Start with the menu.

Check:

• which tools are visible
• which ones are internal-only
• whether descriptions overlap
• whether task-level routing exists
• whether you have tracing

A lot of “model stupidity” is really tool-surface sloppiness.

Once I started looking at agent failures this way, debugging got much less mystical.

Less begging GPT-5 to read my mind.

More designing a system that gives the model a fair shot.