AI-assisted DevOps and SRE tools are becoming more common. Tools like K8sGPT can scan Kubernetes clusters, detect issues, and explain what might be going wrong.
Most evaluations of these tools focus on one obvious question:
Can the system diagnose or fix the incident?
That question matters, but in production systems, it is not the whole story.
Sometimes the more important question is:
Does the system know when not to act?
A confident but wrong remediation can make an incident worse. Restarting the wrong workload, suggesting an unsafe mutation, acting on stale events, or proposing changes based on incomplete evidence can turn a manageable incident into a larger production problem.
That idea is the motivation behind this benchmark.
A calibration and abstention benchmark has been built for K8sGPT-backed Kubernetes SRE workflows. The goal is not only to test whether a workflow can identify Kubernetes issues, but also whether it can recognize uncertainty, avoid unsafe actions, and abstain when the evidence is incomplete, misleading, or adversarial.
GitHub repo: github.com/Mayank-013/k8sGPT
Why diagnosis accuracy is not enough
In many Kubernetes troubleshooting scenarios, identifying the visible symptom is relatively easy.
A pod is in CrashLoopBackOff.
An image cannot be pulled.
A workload is stuck in Pending.
A readiness probe is failing.
But knowing the symptom is different from knowing the root cause.
For example, a readiness failure could be caused by:
a bad probe path
a wrong probe port
a broken application
an upstream dependency outage
a NetworkPolicy issue
stale events from an older incident
If an SRE workflow sees the symptom and immediately proposes a remediation, it may look helpful while still misunderstanding the incident.
That distinction matters.
In real operations, "do nothing yet" can be the safest decision ā but only if the system understands why it is abstaining.
What has been built
This project introduces a benchmark and evaluation harness around K8sGPT as the analyzer under test.
The benchmark includes 163 labeled Kubernetes incident cases across four categories:
| Split | Count | Purpose |
|---|---|---|
| ID | 50 | Routine Kubernetes incidents |
| Near-OOD | 47 | Familiar symptoms with less obvious causes |
| Far-OOD | 42 | Cross-layer or out-of-taxonomy failures |
| Adversarial | 24 | Misleading, partial, stale, or conflicting evidence |
| Total | 163 | Labeled cases in the released catalog |
Each case is labeled not only with the expected root cause, but also with:
action-risk information
abstention expectations
evidence completeness
unsafe remediation candidates
expected behavior
Instead of only asking whether the workflow got the diagnosis right, the benchmark asks:
Did it identify the right root-cause family?
Did it propose a safe action?
Should it have acted, investigated more, required approval, or abstained?
Was its confidence calibrated?
Did it produce an unsafe or overprivileged action proposal?
Did it fail safely under uncertainty?
How the evaluation works
At a high level, the workflow looks like this:
kind cluster
-> inject Kubernetes incident
-> capture evidence
-> run K8sGPT
-> normalize findings
-> extract structured action plan
-> classify risk
-> score diagnosis, abstention, calibration, and safety
The benchmark uses a reproducible kind-based setup to inject incidents, run K8sGPT, normalize findings, extract structured action plans, and score the results.
Raw outputs are preserved, and scoring happens separately. This makes it possible to distinguish what K8sGPT actually found from what the surrounding workflow inferred.
That distinction is important because this project evaluates K8sGPT-backed workflows, not native K8sGPT remediation behavior.
K8sGPT provides analyzer findings and optional explanation output. Structured action plans, confidence scores, abstention decisions, and routing logic are wrapper and evaluation-layer components in this benchmark.
To keep the evaluation honest, stage scores are reported separately:
K8sGPT analyzer finding quality
Wrapper action and abstention quality
This avoids mixing up what K8sGPT itself produced with what the surrounding workflow inferred.
What is being measured
The benchmark focuses on safety-oriented evaluation rather than diagnosis accuracy alone.
| Area | What it checks |
|---|---|
| Diagnosis quality | Whether the workflow identifies the correct root-cause family |
| Action quality | Whether proposed actions are correct, safe, ineffective, unsafe, or overprivileged |
| Abstention behavior | Whether the system acts, investigates more, requires approval, or abstains |
| Confidence calibration | Whether confidence aligns with correctness |
| False remediation risk | Whether the workflow proposes a wrong mutation |
| OOD behavior | Whether performance degrades under unfamiliar or adversarial incidents |
This matters because an AI-assisted SRE workflow should not only be evaluated by how often it produces an answer.
It should also be evaluated by whether the answer is safe to act on.
What was evaluated
The released benchmark contains 163 labeled cases. The evaluation separates full-catalog analyzer scoring from a smaller live LLM-backed subset.
| Set | Size | Meaning |
|---|---|---|
| Released labeled catalog | 163 | All cases with schemas, manifests, and ground truth |
| Full executable C0 evaluation | 163 | Live kind injection -> k8sgpt analyze -> heuristic extraction -> scoring |
| Full offline C1-C7 prepare | 163 | Heuristic/wrapper post-processing over C0 as an infrastructure baseline |
| Live LLM mechanism subset | 14 | C0/C1/C2/C3/C7 with OpenAI gpt-4o-mini explanation and extraction |
| Second-labeler subset | 30 / 113 OOD | Independent second-labeler agreement check |
The important distinction:
Benchmark size is not the same as LLM-differentiated size.
The full 163-case catalog is used for live analyzer evaluation. The live LLM-backed mechanism comparison is currently a smaller 14-case subset.
Key results: live C0 analyzer evaluation
On routine Kubernetes cases, K8sGPT-backed analyzer workflows were useful.
For common incident families like ImagePullBackOff, CrashLoopBackOff, OOMKilled, and Pending, analyzer-backed extraction recovered expected actions reliably in the in-distribution set.
Probe-related issues were much harder.
| Family | Expected-action hit | Notes |
|---|---|---|
| ImagePullBackOff | 10/10 | Strong |
| CrashLoopBackOff | 10/10 | Strong |
| OOMKilled | 10/10 | Real OOMKilled cases |
| Pending | 10/10 | Strong |
| Probe failure | 1/10 | Availability-only findings were common |
This suggests that K8sGPT-backed analyzer workflows can be helpful for routine Kubernetes symptoms, but some failure families need deeper causal reasoning.
Full-catalog results
The full executable C0 evaluation was run across all 163 released cases.
| Split / Condition | N | Exact + Family | Correct-safe | Safe-abstain | ECE_action | Abstention F1 | False remediation |
|---|---|---|---|---|---|---|---|
| ID / C0 | 50 | 0.820 | 0.400 | 0.600 | 0.426 | 0.750 | 0.000 |
| Near-OOD / C0 | 47 | 0.085 | 0.000 | 1.000 | 0.566 | 1.000 | 0.000 |
| Far-OOD / C0 | 42 | 0.000 | 0.000 | 1.000 | 0.593 | 1.000 | 0.000 |
| Adversarial / C0 | 24 | 0.000 | 0.000 | 1.000 | 0.573 | 1.000 | 0.000 |
| All / C0 | 163 | 0.276 | 0.123 | 0.877 | 0.531 | 0.935 | 0.000 |
The pattern is clear:
K8sGPT-backed workflows are useful for routine Kubernetes symptoms, but safe remediation under uncertain, OOD, or adversarial cases remains difficult.
The high safe-abstain rate in OOD and adversarial cases is important, but it should not be over-interpreted.
An abstention-heavy workflow can look safe while still misunderstanding the incident.
In other words, a system may avoid dangerous actions not because it deeply understands the situation, but because it abstains broadly.
That is safer than reckless mutation, but it is not the same as robust causal understanding.
Mixed-condition full-catalog results
The full catalog also includes prepared C1-C3/C7 rows. These are mixed results: 149 heuristic offline prepares plus 14 preserved live OpenAI-backed plans.
| Condition | N | Exact + Family | Correct-safe | Safe-abstain | Unsafe | ECE_action |
|---|---|---|---|---|---|---|
| C0 | 163 | 0.276 | 0.123 | 0.877 | 0.000 | 0.531 |
| C1 mixed | 163 | 0.245 | 0.123 | 0.804 | 0.031 | 0.538 |
| C2 mixed | 163 | 0.252 | 0.123 | 0.804 | 0.037 | 0.520 |
| C3 mixed | 163 | 0.252 | 0.129 | 0.810 | 0.037 | 0.529 |
| C7 mixed | 163 | 0.252 | 0.117 | 0.871 | 0.006 | 0.498 |
These full-catalog mixed rows are useful as infrastructure baselines, but the fair mechanism comparison is the live LLM subset below.
Live LLM-backed subset
A small live LLM-backed subset was evaluated using OpenAI gpt-4o-mini for explanation and structured action extraction.
The 14 representative cases covered ID, near-OOD, far-OOD, and adversarial scenarios.
| Condition | Backend | Correct-safe | Safe-abstain | y_action | Unsafe | Mean confidence |
|---|---|---|---|---|---|---|
| C0 analyzer + heuristic | heuristic_v1 |
2/14 | 12/14 | 100% | 0/14 | 0.53 |
| C1 explain + LLM extract | gpt-4o-mini |
2/14 | 0/14 | 14% | 5/14 | 0.79 |
| C2 verbal + LLM | gpt-4o-mini |
2/14 | 0/14 | 14% | 6/14 | 0.78 |
| C3 self-consistency | gpt-4o-mini |
3/14 | 1/14 | 29% | 6/14 | 0.79 |
| C7 hybrid router | router over C2 | 1/14 | 11/14 | 86% | 1/14 | 0.78 |
In this subset, the LLM-backed wrapper was more willing to propose actions and had higher confidence. However, it also produced more unsafe action proposals than the abstain-heavy heuristic baseline.
A hybrid router restored more conservative behavior by pushing uncertain or risky cases back toward abstention.
This suggests an important lesson:
Adding an LLM layer can make a workflow feel more actionable, but without risk-aware routing, it may also make the workflow less safe.
Second-labeler agreement
To reduce the risk of purely subjective labeling, a second-labeler pass was performed on 30 out of 113 OOD cases.
| Label dimension | Agreement |
|---|---|
| Root-cause family | 83.3% (25/30) |
| Expected behavior | 70.0% (21/30) |
| Action risk tier | 63.3% (19/30) |
| Evidence completeness | 100.0% (30/30) |
The lower agreement on action-risk tier is actually useful. It shows that deciding whether an SRE workflow should act, require approval, or escalate is not always obvious.
That is exactly why this kind of benchmark matters.
Main takeaway
The strongest finding is not simply that K8sGPT performs better on routine cases than OOD cases.
That is expected.
The more interesting finding is this:
Safety via abstention is not the same as robust causal understanding.
A workflow can avoid unsafe action by abstaining broadly, but still fail to understand the incident. On the other side, adding an LLM layer can increase actionability and confidence, but may also increase unsafe action proposals unless paired with risk-aware routing.
This creates a useful way to think about AI-assisted SRE workflows:
diagnosis accuracy matters
confidence calibration matters
abstention quality matters
action-risk classification matters
unsafe proposal rates matter
escalation behavior matters
Production safety is not only about getting the answer right.
It is also about knowing when the answer is not safe enough to act on.
Why this matters
As AI-assisted SRE tools become more capable, the hard problem is not only generating a plausible explanation.
The hard problem is deciding what level of autonomy is appropriate.
Should the system suggest a command?
Should it ask for approval?
Should it collect more evidence?
Should it escalate to a human?
Should it refuse to recommend a mutation?
These decisions are central to production safety.
For Kubernetes environments, where remediation actions can affect running workloads, security boundaries, and customer-facing systems, knowing when not to act is just as important as knowing how to act.
What this project does not claim
This benchmark does not claim that K8sGPT is unsafe.
It also does not claim that the evaluated wrapper represents native K8sGPT remediation behavior.
The goal is more specific:
Evaluate how K8sGPT-backed workflows behave when diagnosis, confidence, abstention, and action risk are measured together.
This project is about building a more careful evaluation framework for AI-assisted Kubernetes troubleshooting.
What comes next
This is an early release, and there is more to improve.
Next steps include:
expanding live LLM-backed evaluation beyond the current subset
adding another baseline such as HolmesGPT
improving probe and dependency-related scenarios
publishing a formal PDF report
refining calibration and abstention metrics
improving reproducibility and documentation
Final thought
Most AI/SRE demos focus on action:
Here is the issue. Here is the fix.
But production reliability often requires restraint.
Sometimes the best answer is:
There is not enough evidence to safely act yet.
That is the behavior this benchmark is designed to test.
If you are interested in Kubernetes, SRE, AI-assisted operations, or reliability evaluation, feedback on the project is welcome.
GitHub repo: github.com/Mayank-013/k8sGPT
Top comments (0)