Disclosure up front: I work on Radar, the open-source Kubernetes UI whose MCP server is one side of this benchmark. The method is published, so you can check the work.
Everyone is plugging AI agents into Kubernetes right now, and the default move is to give the agent a shell and let it run kubectl. It works. The agent gets there. But nobody seemed to have measured what that approach actually costs, so we built a harness and did.
The setup
- 52 fault-injection scenarios on a live EKS cluster: crashloops, misconfigurations, resource pressure, broken rollouts, and the messier indirect failures where the symptom is far from the cause.
- One model on both sides: Claude Sonnet 4.6.
- Two conditions. Condition A: the agent gets a shell and raw kubectl. Condition B: the same agent connects through Radar's MCP server.
- Same prompts, same success criteria, scored on whether the agent found the actual root cause.
The only variable is how the agent sees the cluster.
The results
Per-trial averages across all 52 scenarios:
| Metric (per trial) | kubectl | Radar MCP | Change |
|---|---|---|---|
| Tool calls | 45.8 | 11.1 | 76% fewer |
| Input tokens | 4.9M | 2.3M | 53% fewer |
| Output tokens | 3,040 | 1,039 | 66% fewer |
| Agent time | 334s | 169s | 49% faster |
| Pass rate | 77.6% | 80.8% | small increase (+3.2pp) |
| Diagnostic score | 0.765 | 0.862 | small increase (+0.097) |
The headline is the efficiency column. The accuracy rows are there to show the speedup wasn't bought with correctness: it nudged up rather than degrading.
Why the gap exists: it's the context, not the protocol
The tempting reading is "MCP beats shell access." That's not quite it, and it matters, because plenty of Kubernetes MCP servers are essentially a proxy to kubectl: the same raw output over a newer transport. Wrap kubectl in MCP tools and I'd expect numbers close to the kubectl column.
Watch a kubectl-driven agent work and the cost is obvious. It runs get pods, then describe, then logs, then get events, then describe on the owner, and after every call it re-derives the same structure from text: which replicaset belongs to which deployment, which service fronts which pods, what changed before things went bad. That structure never persists between calls. The agent pays to rebuild it, in tokens and in round trips, over and over.
Radar's MCP server hands the agent that structure as ready context. Two pieces do most of the work:
- The resource graph. Ownership chains, service routing, what's connected to what. One tool call returns the neighborhood of a failing resource instead of five calls' worth of raw output to correlate.
- The change timeline. What was deployed, scaled, restarted, or rolled back, and when. Most diagnoses are some form of "what changed?", and giving the agent the answer as data beats making it reconstruct the sequence from events and log timestamps.
That's the whole trick: better data in a better format. 45.8 tool calls collapse to 11.1 because most of those calls existed to rebuild context the server can simply provide.
Where the gap was widest
Not on the easy faults. A pod stuck in CrashLoopBackOff with an obvious error in its logs gets diagnosed either way, and kubectl does fine.
The gap opened on indirect, cross-cutting failures: the ones where the failing thing isn't the broken thing, and the trail crosses resource boundaries. Those are exactly the cases where a human relies on knowing the topology and the recent history, and the agent behaves the same way. With the graph and the timeline in context it followed the chain; with raw output it burned calls wandering.
Honest caveats
- The accuracy gain is small. +3.2 points of pass rate is a nudge, not a leap. If you want this benchmark to prove agents get dramatically smarter with MCP, it doesn't. It proves they get dramatically cheaper and faster at the same quality.
- One model. We ran Claude Sonnet 4.6. The mechanism (ready structure is cheaper to reason over than raw text) shouldn't be model-specific, but we haven't measured others yet. If you run it on GPT, Gemini, or a local model, I'd genuinely like to see the numbers.
- 52 scenarios. Big enough to see the pattern, small enough to inspect by hand. We'll grow it, and if there's a fault class you think is missing, tell me.
Try it or reproduce it
The full method is in the benchmark write-up this post is adapted from, and the background on how the MCP server builds live cluster context is here: MCP for Kubernetes: live cluster context for AI tools.
Radar is a single Go binary, Apache-2.0, no account, nothing installed in the cluster. It reads your kubeconfig and respects your RBAC:
curl -fsSL https://get.radarhq.io | sh && radar
Repo: github.com/skyhook-io/radar. The MCP server is in the box; point your own agent at it and see if your numbers look like ours.
Top comments (5)
This is one of the better discussions I’ve seen around AI + Kubernetes because the focus is on improving the agent’s context rather than just exposing more tools.
One thing I’d still love to see is a fully reproducible benchmark package: scenario list, prompts, raw agent trajectories, and repeated runs with confidence intervals. The efficiency gains make intuitive sense, but publishing the complete artifacts would make the results even stronger and easier for the community to validate. Great work.
Hey Mustafa, original author here. Totally agree - it's a bit difficult to reproduce perfectly because the benchmarks aren't set up that way right now. We used what seemed like the best open benchmark for this in SREGym but had to do some work around it to automate running agents side by side, tracking the metrics etc.
And SREGym itself is constantly evolving, changing both the harness and the scenarios being tested.
Eventually we'd like to get to the point that this does become easily reproducible. Until then, happy to share some more of the raw data if you're interested
Thanks for the detailed reply. I appreciate the transparency. I think publishing the automation layer around SREGym would already be a big step, even if the benchmark evolves over time. The scenario list, prompts, trajectories, and multiple-run results would let others validate the conclusions without having to reproduce the exact numbers.
SREGym is already becoming a useful baseline for agent evaluation, so having comparable artifacts across projects would be valuable for the whole community.
References:
The efficiency column is convincing, and "it got cheaper, not smarter" is the honest read. I wish more benchmark posts drew that line.
One note on the accuracy rows, because they're stronger than they look. 77.6% vs 80.8% reads as noise if you treat it as two independent proportions (back-of-envelope, a two-proportion SE on 52 trials is ~8pp, so +3.2pp is under 1 SE). But this is a paired design: same 52 scenarios, same model, only the view changes. So the honest test is McNemar on the discordant scenarios, not the marginals. Pairing removes scenario difficulty, which makes it more powerful than the unpaired delta suggests. It could surface a real small shift, or confirm accuracy is genuinely flat.
The direction of the flips is the part a flat marginal hides. If graph+timeline fixes 6 indirect-failure cases but the abstraction costs you 3 easy ones where the shell's raw log line was the tell, net is +3 and looks like a nudge, when it's actually a real trade. The discordant cells are your "where the gap was widest" section made quantitative, and they're the only thing that answers whether ready context ever removes a signal the raw output had.
One caveat that feeds the same test: with ~1 trial per scenario, some flips are just agent nondeterminism rather than the condition, so a couple of repeats per scenario would make those discordant counts trustworthy.
The 76% drop in tool calls tracks with what I have seen: giving an agent a change timeline removes most of the blind kubectl get loops it does just to rebuild state. The resource graph is doing the work a human does mentally when they say "what changed recently." Did you measure how many of the kubectl-side calls were pure re-discovery versus actual remediation steps?