Hey!
This is the third post about Xonta, orchestrator that decides how much harm an AI agent can be trusted with. In the earlier parts agents refunded customers and hunted for company phone numbers. This time I took the real MCP catalog from Glama https://glama.ai and looked at how agents make sense of other people's tools. Spoiler - a cheap agent with tips from the orchestrator beat the expensive model, and Xonta itself got a couple of honest kicks along the way and came out better.
Quick note first. I didn't connect to a single server or call a single tool. Everything is built on the descriptions, schemas and hints that Glama serves through its open API. Thanks for the catalog, Glama.
Why bother
Picture an agent that just plugged in someone else's MCP server and sees a tool called update_campaign. It has a second to decide whether it can call it without asking. Get it wrong one way and you bother a human for nothing, no big deal. Get it wrong the other way and you've deleted data, sent an email or made a payment. That one hurts, and there's no undo.
The Glama catalog has a nice feature. Lots of publishers tag their own tools with readOnlyHint and destructiveHint, so you know whether a tool reads or changes things and whether that can be rolled back. That's a ready-made reference to check agents against. I wanted to know three things. Can you trust an agent that judges a tool by its description. What does Xonta see when the agent works through it. And the big one, can Xonta do more than catch mistakes, can it make agents work better and cheaper.
What's in the catalog
I pulled every remote server, 998 of them, with 19,944 tools. For 95.9% of the tools the publisher set hints, and they're real, not defaults filled in by the catalog (I checked, the fields and values vary). Self-hosted servers had to stay out, the API just doesn't list their tools.
About 10% of tools I flagged as "need attention". Not "offenders", just a reason to take a closer look. The most common case is an irreversible action that the description doesn't mention at all, that's 9.1%. Another 120 hits came from a "looks like a hidden instruction to the model" rule, but that's a heuristic and every hit needs a human look, so no big claims there. Funny enough, Glama's own quality score agrees. Among servers scored 4-5, 26% have a flagged tool, among those scored 2-4 it's 51%. I'm not publishing a per-server list, that wouldn't be fair.
Stage one, one agent against the catalog
First I compared two ways of understanding a tool. The Xonta scanner looks only at the name, which is how Gate works today. A GigaChat agent reads the description.
The scanner can barely tell reversible from irreversible, and worse, it takes a quarter of irreversible tools for harmless reads. That's because names like save_item, resolve_task or end_call don't carry the usual action word. The agent is the opposite. It almost never mistakes a dangerous tool for a read, just 2 out of 100, but it calls half the irreversible ones reversible. They fail in different ways, and if you take the stricter of the two answers you catch 80% of irreversible tools. For Gate the takeaway is simple, the name alone isn't enough.
Here's what Xonta did along the way, and I wouldn't have seen any of it without it. Before working, the agent took an exam, scored 60% and got minimal rights. During the work, every hundred rows the orchestrator picked some at random and handed them to another agent owned by someone else for a re-check. Our auditor called 97% of tools "no concerns", and in the second run the re-check didn't confirm 12 of its 42 "all good" calls. That's significant (p = 0.0006), so the orchestrator paused the job and pinged the owner. On top of that the agent learned from its mistakes and threw out more than ten rules over two runs, several of them the same "if the name says write, it's irreversible". The data says the opposite, and not one rule passed the check.
And one instructive story. In the very first run a third of the model's answers couldn't be parsed at all. Turned out GigaChat was putting the tool name into the label field instead of an answer, just because the input had a tool field right next to it. I renamed the field to effect, listed the allowed values, and unparsed answers dropped to 0.3%. If I hadn't counted those answers separately, that third would have quietly gone into "the agent is unsure" and the conclusions would have drifted.
Stage two, a team of three and the first faceplant
Next I built a team. fast looks only at the name and the start of the description, it's cheap. careful reads the description and parameters and costs 2.3 times as much. dual does what careful does and checks itself against the scanner, picking the stricter answer.
dual has its own trick, "stricter of two", and careful has another, "don't call the model if the name obviously says it's a read". The Xonta orchestrator tested both in pairs on fast's tasks, meaning the same row without the trick and with it. And here's the faceplant. On version 0.21 both tricks got "no effect".
I looked closer and realised the bug was in Xonta itself. "Stricter of two" made 10 rows better and 13 worse. But "better" meant caught deletions and payments, where a mistake costs 6-10 units, and "worse" meant extra caution that costs one. Overall harm dropped by 16%, but the orchestrator counted pairs as right or wrong and didn't see the cost of a mistake. The second trick broke nothing and saved 11% of tokens, but Xonta simply had no outcome like "same quality, just cheaper". And it never advised a specialist child because it only looked at accuracy gaps.
So I fixed Xonta, that's version 0.22. When an agent reports the cost of a mistake, tricks are compared by harm. There's a new outcome, "same, but cheaper", for when no pair got worse and the cost went down noticeably. And the advice to spawn a child now counts harm, not only accuracy.
After the fix the orchestrator passed it all on by itself
On 0.22 the orchestrator proved both tricks by itself and passed them to the cheap agent. Two runs in a row, so it's not luck. Here are the numbers on the same 240 tools.
In plain words, after the orchestrator's tips the cheap agent catches dangerous tools as well as the most careful one on the team, at 2.5 times less cost. The price is a couple of points of overall accuracy, because it plays it safe more often and says "irreversible" where things could be rolled back. But for "call it without asking or not" that's the right trade, an extra question to a human is cheaper than a deleted database.
Stage three, calling in the senior model
That left the question about children. The "spawn a specialist" advice fires when someone on the team is clearly stronger in a weak category. Fine, I thought, I'll add a fourth auditor on GigaChat-2-Max, it's obviously smarter. Same approach as careful, bigger model.
For me this is the key moment of the whole story. The senior model came out the most accurate overall and at the same time the most dangerous. It finds irreversible tools worst of all, does the most harm and runs twice as slow. It just says "this can be undone" more often. If we picked the agent by percentage of right answers, we'd pick exactly this one. By harm it's the other way round, and the winner is the cheap agent with a trick from the orchestrator.
By the way, the orchestrator never advised a child. Not with the full 150-task exam, not with the senior model. In the weak categories all four agents score about the same on the exam, and every time Xonta answers honestly - hand those tools to a human for now, there's nothing to justify a specialist. I like that it doesn't invent a child for a pretty picture, but it also means I couldn't test the mechanism on this data. In the stand's theme runs it works, there's something to lean on there.
One more small thing that made me happy. In the fourth run the "don't call the model" trick didn't pass as "same, but cheaper", though it had passed twice before. I dug in and found one tool where the trick answered "read only" and the publisher had marked a change. The mistake cost for that pair went from 1 to 4, so "same quality" was no longer true. The safeguard did exactly what it's supposed to.
What Xonta caught in the end
Put it all together and this is what I wouldn't have seen without the orchestrator.
- The agent rates itself too high, every fourth or fifth "all good" fails an independent check.
- The agent confidently proposes a wrong rule, and more than once.
- A third of a model's answers can turn out to be junk because of one word in the prompt.
- The most accurate model can easily be the most dangerous one for a specific job.
- One agent's good trick can be passed to another and shown to help, not by eye but in pairs on the same tasks.
What I had to fix in Xonta itself
The most valuable part of experiments like this is when your own system breaks. Three things broke here. Trick scoring didn't see the cost of a mistake and had no "cheaper at the same quality" outcome, version 0.22 closed that. The long-job signal showed the numbers from the last check instead of the total that actually triggered it. And the "keep the child" advice didn't say why it was keeping it, accuracy or less harm.
Plus one open item that a commenter on the previous post pointed out. A trick's verdict doesn't record which scoring rules its pairs were counted under. I changed scoring twice, and if an orchestrator's book lives across versions, old and new pairs can get mixed. In these runs I kept them apart by hand, just wiping the book before each run. That's not a mechanism, so stamping the scoring version on every verdict is the next fix.
Where's the catch
Honest limits. All models are from one family, GigaChat, one or two runs per setup, and overall accuracy moves 2-4 points between runs, so comparisons inside one run are more reliable than across runs. The reference is publisher hints, not real tool behaviour, and the hints themselves are sometimes debatable (update_meal is marked destructive and send_message reversible, go figure). The mistake costs are my own scale of 10, 6, 4, 1, and with another scale the verdict on "stricter of two" could change. And self-hosted servers stayed unchecked.
The code is open, it's all in the Xonta repo under research/glama, reports under results. If you have your own tool catalog or an idea for what else to run agents on, drop it in the comments, I'm genuinely curious.
Top comments (0)