Everyone wants smarter AI agents.
We compare models.
We tweak prompts.
We experiment with system instructions.
But after spending months working with MCP tools like Playwright and Figma, I realized something surprising:
The biggest bottleneck wasn't the model. It was the data we were feeding it.
The Hidden Problem
Imagine asking an AI agent a simple question:
"Is the Login button visible?"
That sounds like a tiny task.
Yet many browser tools respond with something closer to this:
- the complete accessibility tree
- every DOM node
- dozens of element attributes
- hidden elements
- unrelated page metadata
- viewport information
- screenshot metadata
- repeated values from previous snapshots
Instead of receiving the few bytes actually needed to answer the question, the model often receives hundreds of kilobytes.
The same thing happens with design tools.
Ask for the color of a button in Figma, and you might receive an entire page hierarchy containing thousands of nodes.
The AI only needed one component.
It received the entire design.
Bigger Context Doesn't Mean Better Results
There's a common assumption that giving an LLM more information always improves accuracy.
In practice, the opposite often happens.
Large tool responses introduce:
- irrelevant context
- duplicated information
- longer reasoning time
- higher token usage
- increased latency
The model spends part of its context window understanding data that has nothing to do with the user's request.
That's not intelligence.
That's overhead.
The Snapshot Problem
Browser snapshots are one of the biggest contributors.
Imagine an agent performing this workflow:
- Open page
- Click Login
- Wait
- Fill email
- Fill password
- Click Continue
After every action, the browser sends another complete snapshot.
But what actually changed?
Usually only a handful of nodes.
The rest of the page is identical.
Yet the entire snapshot is transmitted again.
It's similar to downloading an entire Git repository after changing a single line of code.
We solved this problem years ago in software engineering with incremental updates and diffs.
Many AI tools still don't.
Noise Is Expensive
People usually calculate the cost of AI using model pricing.
For example:
- GPT costs X per million tokens.
- Claude costs Y per million tokens.
But those calculations ignore something important.
Every unnecessary byte produced by a tool also becomes part of the model's input.
If your tools generate 10× more data than necessary, you're effectively paying for that noise.
And you're paying for it repeatedly throughout the conversation.
The model has no way to know which information is useful until it has already processed it.
Smarter Tools Beat Smarter Prompts
Prompt engineering gets a lot of attention.
Tool engineering gets almost none.
But if your agent only receives relevant information:
- responses become faster
- token usage drops
- context stays cleaner
- reasoning becomes more focused
Sometimes the best optimization isn't changing the prompt.
It's changing what reaches the model in the first place.
Lessons Learned
Working with browser automation and design MCP servers taught me that most optimization opportunities exist before the model even starts thinking.
Instead of asking:
"How can I make the model smarter?"
a better question might be:
"Why am I sending it all this data?"
Once you start looking at tool responses instead of prompts, you'll often discover that most of the context window is occupied by information nobody actually needed.
Final Thoughts
Large language models are becoming increasingly capable.
But they're still constrained by the quality of their inputs.
As AI agents become more common, I think we'll spend less time optimizing prompts and more time optimizing the tools that feed those prompts.
Because sometimes the fastest, cheapest, and most accurate model isn't the one with the largest context window.
It's the one that never had to read hundreds of unnecessary kilobytes in the first place.
Have you ever inspected the raw responses your AI tools send to the model?
The results might surprise you.
Top comments (1)
Strong point. A response budget should be part of the tool contract, not something the model has to recover from after the fact.
The pattern I like is: return the smallest useful result first, include stable identifiers, and make expansion explicit. For browser/Figma/database tools that might mean summary -> targeted node/query result -> optional detail fetch, instead of full snapshot every turn.
Diffs are also underrated here. If the agent already saw the previous state, sending "what changed" is usually more useful than sending the world again. It cuts tokens, but more importantly it reduces the chance that the model reasons over stale or irrelevant context.