We plugged in 13 vendor servers and 300 tools, and they made our agent slow and dumb. Then we wrote one of our own with 30 tools, and the expensive model got cheaper.
The first time Juno handled a real pager, I watched it the way you watch a toddler carry a full glass of water across a room.
Juno is the AI bot we built for engineers in Slack. Someone gets paged at 2am because a service is throwing errors, and instead of opening six dashboards they ask Juno. I wrote about its system prompt in an earlier post. This one is about its tools, which turned out to be the bigger lesson.
That first pager took Juno about fifteen minutes. It got the right answer. It also queried the metrics system eleven times, listed the pods twice, and spent most of the quarter hour gluing JSON together.
We ran it on one of the cheaper models, because the expensive one would have cost real money per incident. Today Juno runs on the most expensive model we can buy. The same kind of incident takes a couple of minutes, and the thread costs less than it did.
Nothing about the model made that happen. We changed what the tools are.
Written with help from an AI assistant, from my own notes and test results.
The free lunch
When you build an agent today, the fastest way to give it powers is MCP, the protocol that lets a model call other systems. Every system you already have ships a server you can plug in: PagerDuty, Grafana, Jira, GitHub, the metrics database, the log platform. Ten minutes of config and your agent can “do anything”.
So that’s what we did. Our first Juno was a coding-agent SDK pointed at our internal MCP gateway. Thirteen integrations. About three hundred and ten tools, plus a shell for kubectl.
The metrics server alone exposed export_csv, export_native, reset_cache and build_info. Nobody on our team has ever asked a bot to export metrics as CSV. But the tool was there, with its description, and the model read it before every single step.
Three hundred tools, zero answers
Watch an incident trace from that era and the real problem is obvious. Someone asks: “is the checkout service healthy in production?”
A human on call knows what that means: are the pods up, are they restarting, is the error rate elevated, is latency up, did something deploy recently, what’s in the logs. One question, six facts.
Juno had no tool that knew what a service was. It had a tool that lists pods. A tool that describes one pod. A tool that runs one metrics query. A tool that fetches logs for one container. So it did what the tools allowed: list the pods, pick one, describe it, query the error rate, query it again with a different label, fetch logs, forget to check latency, and answer.
The tools answered the systems’ questions. Nobody had built one that answered the person’s.
That was the whole fifteen minutes. Not thinking. Plumbing. And every step was a chance to stop early, pick the wrong pod, or misread a label, which happened often enough that nobody trusted the result.
Our own tools had the same disease
Here is the part that convinced me it wasn’t the vendors’ fault.
By September we had replaced most of the MCP servers with tools we wrote ourselves. Reads against our clusters, our metrics, our config repos. Fast, scoped, ours. And the first version of our incident flow still took 29 tool calls and eight minutes on a real alert.
We timed where the eight minutes went. 461 seconds was the model writing. 21 seconds was tools running.
The tools were fast. The model was spending its time deciding which of our reads to call next and stitching the results together, because our reads were still shaped like the APIs underneath them: find the alert rule, run a metric query, list the resources, read the file. Every investigation opened with the same five reads, assembled by hand over four or five turns.
So we folded those five reads into one call. Same alert, same model, same prompt.
That alert, it turned out, had never been raised by our cluster at all. It came from an external daily check. Juno had reached that exact conclusion four times before, after 26, 42, 55 and 84 calls. With the bundled read it arrived on call one.
Asking what people actually ask
The real turn came when we stopped looking at our systems and looked at our Slack history.
We pulled ninety days of pages, support tickets and production-channel threads and tagged what people were actually asking for. One question dominated: “is my service working?” It was 38% of all high-urgency pages and more than half of the production channel. After it came “why is my deployment stuck”, “add replicas to X”, and “what changed”.
Not one of those is an integration. Each is a question with a known shape, whose answer comes from four or five systems at once.
So we built a tool for the question. get_service_health takes a service name and an environment and returns, in one call, everything the on-call engineer would have checked: replicas against desired, each unhealthy pod with its restart reason, warning events, the autoscaler, whether the deployment is in sync, request rate, 5xx rate, p99 latency, the service’s own metrics, and the error lines from its log. It also takes a time, because by the time anyone reads a page the blip is usually over, and “healthy right now” is the wrong answer.
In our test suite the health scenarios went from 0 out of 6 passing to 6 out of 6. Nothing else moved. One tool, one question, fixed.
The expensive model got cheaper
One thing we stopped arguing about: how smart an agent is comes from the model, full stop. No prompt makes a model reason better than it can. What you control is how many turns a question takes and what each turn has to do. That is a tool design problem, and it decides whether you can afford the model you actually want.
With three hundred tools the loop was long, so each turn was expensive, so we had to run the cheap model to make Juno viable at all. The cheap model is also the one that stops halfway through a chain of reads. We were paying for a mediocre engineer by the hour.
With question-shaped tools the loop is short. A health question is one call. A “what changed” question is one call. The expensive model does the thing it’s actually good at, reading the facts and deciding, and does it in two or three steps instead of twelve.
So we switched to the top model. Side by side on our test suite it scored 198 to 184 against the previous one, and on change requests it answered first and acted second three times out of three, where the other model managed it once. A full thread now costs between roughly $0.70 and $1.60.
Cost per token went up. Cost per answer went down. The tools decide which one you pay for.
What we gave up
This is not free, and the post would be dishonest if it ended before saying so.
A vendor MCP server costs nothing to add and somebody else maintains it. Our tools are code we own. get_service_health knows that our service-mesh sidecar is a native init container, that our pods carry a specific label, that our autoscaler reports a specific metric. When the platform changes, the tool changes, and that’s our job now.
Those 29 tools are an MCP server we wrote ourselves, and the same server feeds both Juno and the engineers’ own coding agents. The protocol was never the problem. The shape of the tools was: they were built for a system’s API, not for a person’s question. If a vendor shipped a server with get_service_health on it, I’d use it tomorrow.
And the one thing that still bites: the model only knows what a “service” is because we wrote a resolver that reads 41 configuration repos to map every name people use to the thing actually running. That resolver has had more bugs than any tool. Nobody gets to skip it.
If you’re building one of these
Today Juno has 29 tools. The number isn’t the point. Each one answers something a real person has actually asked, and these are the rules that got us there:
- Don’t start from your integrations. Start from what your users actually ask. Count the questions. Build a tool per question, not per API.
- One tool, one object. A service, a rollout, a ticket. A new fact becomes a field on the existing tool, never a new tool.
- One call answers the question. If the model has to chain three reads, it will stop after two. When eleven deployment tools became two, the model made fewer calls and passed more tests.
- Facts, never verdicts. A tool returns data. Ranking and deciding stay with the model, which can read a sentence. A script can only match words.
- Name a tool after what it returns, in the words your users use. Someone asks for “the logs of the service”, so the tool is get_service_logs. The name says what it is; the description says when to use it.
- Time where the seconds go. Ours were 461 seconds of model and 21 of tools. Fast tools don’t fix a slow loop.
- Then buy the good model. Prompts don’t make a model smarter. Short loops make the smart model affordable.
We started with every tool we could plug in and a model we could afford. We ended with the few tools people actually need and the model we actually wanted. The toddler puts the glass down in two minutes now, and it doesn’t spill.
Top comments (0)