Disclosure up front: I work on Radar, the open-source Kubernetes UI whose MCP server is one side of this benchmark. The method is published, so you can check the work.
Everyone is plugging AI agents into Kubernetes right now, and the default move is to give the agent a shell and let it run kubectl. It works. The agent gets there. But nobody seemed to have measured what that approach actually costs, so we built a harness and did.
The setup
- 52 fault-injection scenarios on a live EKS cluster: crashloops, misconfigurations, resource pressure, broken rollouts, and the messier indirect failures where the symptom is far from the cause.
- One model on both sides: Claude Sonnet 4.6.
- Two conditions. Condition A: the agent gets a shell and raw kubectl. Condition B: the same agent connects through Radar's MCP server.
- Same prompts, same success criteria, scored on whether the agent found the actual root cause.
The only variable is how the agent sees the cluster.
The results
Per-trial averages across all 52 scenarios:
| Metric (per trial) | kubectl | Radar MCP | Change |
|---|---|---|---|
| Tool calls | 45.8 | 11.1 | 76% fewer |
| Input tokens | 4.9M | 2.3M | 53% fewer |
| Output tokens | 3,040 | 1,039 | 66% fewer |
| Agent time | 334s | 169s | 49% faster |
| Pass rate | 77.6% | 80.8% | small increase (+3.2pp) |
| Diagnostic score | 0.765 | 0.862 | small increase (+0.097) |
The headline is the efficiency column. The accuracy rows are there to show the speedup wasn't bought with correctness: it nudged up rather than degrading.
Why the gap exists: it's the context, not the protocol
The tempting reading is "MCP beats shell access." That's not quite it, and it matters, because plenty of Kubernetes MCP servers are essentially a proxy to kubectl: the same raw output over a newer transport. Wrap kubectl in MCP tools and I'd expect numbers close to the kubectl column.
Watch a kubectl-driven agent work and the cost is obvious. It runs get pods, then describe, then logs, then get events, then describe on the owner, and after every call it re-derives the same structure from text: which replicaset belongs to which deployment, which service fronts which pods, what changed before things went bad. That structure never persists between calls. The agent pays to rebuild it, in tokens and in round trips, over and over.
Radar's MCP server hands the agent that structure as ready context. Two pieces do most of the work:
- The resource graph. Ownership chains, service routing, what's connected to what. One tool call returns the neighborhood of a failing resource instead of five calls' worth of raw output to correlate.
- The change timeline. What was deployed, scaled, restarted, or rolled back, and when. Most diagnoses are some form of "what changed?", and giving the agent the answer as data beats making it reconstruct the sequence from events and log timestamps.
That's the whole trick: better data in a better format. 45.8 tool calls collapse to 11.1 because most of those calls existed to rebuild context the server can simply provide.
Where the gap was widest
Not on the easy faults. A pod stuck in CrashLoopBackOff with an obvious error in its logs gets diagnosed either way, and kubectl does fine.
The gap opened on indirect, cross-cutting failures: the ones where the failing thing isn't the broken thing, and the trail crosses resource boundaries. Those are exactly the cases where a human relies on knowing the topology and the recent history, and the agent behaves the same way. With the graph and the timeline in context it followed the chain; with raw output it burned calls wandering.
Honest caveats
- The accuracy gain is small. +3.2 points of pass rate is a nudge, not a leap. If you want this benchmark to prove agents get dramatically smarter with MCP, it doesn't. It proves they get dramatically cheaper and faster at the same quality.
- One model. We ran Claude Sonnet 4.6. The mechanism (ready structure is cheaper to reason over than raw text) shouldn't be model-specific, but we haven't measured others yet. If you run it on GPT, Gemini, or a local model, I'd genuinely like to see the numbers.
- 52 scenarios. Big enough to see the pattern, small enough to inspect by hand. We'll grow it, and if there's a fault class you think is missing, tell me.
Try it or reproduce it
The full method is in the benchmark write-up this post is adapted from, and the background on how the MCP server builds live cluster context is here: MCP for Kubernetes: live cluster context for AI tools.
Radar is a single Go binary, Apache-2.0, no account, nothing installed in the cluster. It reads your kubeconfig and respects your RBAC:
curl -fsSL https://get.radarhq.io | sh && radar
Repo: github.com/skyhook-io/radar. The MCP server is in the box; point your own agent at it and see if your numbers look like ours.
Top comments (10)
Really enjoyed this benchmark. As someone exploring AI automation, it's fascinating to see how structured context can dramatically reduce token usage and tool calls without sacrificing accuracy. Great insights!
This is one of the better discussions I’ve seen around AI + Kubernetes because the focus is on improving the agent’s context rather than just exposing more tools.
One thing I’d still love to see is a fully reproducible benchmark package: scenario list, prompts, raw agent trajectories, and repeated runs with confidence intervals. The efficiency gains make intuitive sense, but publishing the complete artifacts would make the results even stronger and easier for the community to validate. Great work.
Hey Mustafa, original author here. Totally agree - it's a bit difficult to reproduce perfectly because the benchmarks aren't set up that way right now. We used what seemed like the best open benchmark for this in SREGym but had to do some work around it to automate running agents side by side, tracking the metrics etc.
And SREGym itself is constantly evolving, changing both the harness and the scenarios being tested.
Eventually we'd like to get to the point that this does become easily reproducible. Until then, happy to share some more of the raw data if you're interested
Thanks for the detailed reply. I appreciate the transparency. I think publishing the automation layer around SREGym would already be a big step, even if the benchmark evolves over time. The scenario list, prompts, trajectories, and multiple-run results would let others validate the conclusions without having to reproduce the exact numbers.
SREGym is already becoming a useful baseline for agent evaluation, so having comparable artifacts across projects would be valuable for the whole community.
References:
Appreciate that the method is published — that's the part that makes this worth reading. What jumps out is that the efficiency gains are huge (76% fewer tool calls, half the time) but the pass rate barely moves, 77.6% to 80.8%. That's actually a really honest result: giving the agent a resource graph and a change timeline mostly buys you cost and speed, not correctness — the hard root-cause cases seem to stay hard either way. Did you get any signal on whether the 3-ish point pass-rate bump came specifically from the change-timeline (seeing what changed and when) versus the topology view? That'd tell you which half of the context is actually load-bearing for the messy indirect failures.
Hi James, so honestly the pass rate here was possibly noise, it's small enough that I wouldn't draw conclusions from it. The efficiency win was clear across the board though.
Topology vs change-timeline - I don't have hard numbers but both contributed, I think topology more but definitely some scenarios get "solved" pretty directly from looking at the change-timeline which gets automatically associated with resources and issues, brought up to the agent's attention at varying rate depedning on the type of change.
We're constantly working on improvements to Radar, for depth and breadth of detection and reasoning capabilities. There's no silver bullet, it's a lot of work to map out many different scenarios and do it carefully enough so as not to generate noise, and not overfit benchmarks.
The way SREGym is built, it takes a very long time to run, analyze the results, understand what radar does well and where we have a way to go.
We're going to release some followup to this with some corrections and some pretty exciting updates on latest Radar version and newer model :)
The disclosure up front and the published method put this above most vendor benchmarks, and the honest framing of the accuracy rows deserves credit too: at 52 scenarios, +3.2 points is inside the noise, and you did not pretend otherwise. If each scenario ran once, 80.8 percent is 42 of 52, and the 95 percent Wilson interval spans roughly 68 to 89 percent, which comfortably covers the kubectl condition. One thing the table implies: 77.6 percent is not an integer count out of 52, so each fault presumably ran more than once, or some runs were excluded. That raises the effective n, so treat my 42-of-52 interval as the single-run floor. The efficiency columns are a different story. A 76 percent drop in tool calls and half the wall time are effect sizes that survive any reasonable interval at this n, so the headline points at the right column. One cheap addition would sharpen the accuracy claim: both conditions ran the same 52 faults, so the data is paired, and the discordant scenarios (the ones where exactly one condition found the root cause) are the entire accuracy signal. McNemar on those few cases is the right test, and my guess is it lands on no detectable difference, which is fine, because that is not the claim the benchmark needs.
The efficiency column is convincing, and "it got cheaper, not smarter" is the honest read. I wish more benchmark posts drew that line.
One note on the accuracy rows, because they're stronger than they look. 77.6% vs 80.8% reads as noise if you treat it as two independent proportions (back-of-envelope, a two-proportion SE on 52 trials is ~8pp, so +3.2pp is under 1 SE). But this is a paired design: same 52 scenarios, same model, only the view changes. So the honest test is McNemar on the discordant scenarios, not the marginals. Pairing removes scenario difficulty, which makes it more powerful than the unpaired delta suggests. It could surface a real small shift, or confirm accuracy is genuinely flat.
The direction of the flips is the part a flat marginal hides. If graph+timeline fixes 6 indirect-failure cases but the abstraction costs you 3 easy ones where the shell's raw log line was the tell, net is +3 and looks like a nudge, when it's actually a real trade. The discordant cells are your "where the gap was widest" section made quantitative, and they're the only thing that answers whether ready context ever removes a signal the raw output had.
One caveat that feeds the same test: with ~1 trial per scenario, some flips are just agent nondeterminism rather than the condition, so a couple of repeats per scenario would make those discordant counts trustworthy.
The key point is that the gain came from better context shape, not just a transport swap. Giving an agent typed cluster affordances instead of raw shell output changes how much time it spends reconstructing intent from text.
That also makes the benchmark more useful than the usual "it felt faster" claim. Once you can line up task success with the exact tool path, retries, and intermediate observations, you can explain where the efficiency came from instead of just reporting the final number.
This is very aligned with why we built agent-inspect: local-first execution traces make it much easier to debug why an agent chose one tool sequence over another. Did your harness keep those step-level traces for later analysis, or mostly aggregate metrics?
Thanks Raju - harness did keep step level traces, though it's some work to collect and analyze. Will take a look at agent-inspect!