AI agents aren’t ready for IT operations yet and now there’s a benchmark that proves it

There’s a lot of momentum right now behind building and deploying AI agents to automate critical IT tasks, often without really assessing how efficient they are in real-world conditions.

Assessing here doesn’t mean testing in a controlled sandbox, it means running the agent across many real-time scenarios and measuring its actual performance and efficiency. That’s why benchmarking agents matters, the same way benchmarking LLMs does. That’s where ITBench comes in.

What ITBench actually measures ?

ITBench is a first-of-its-kind framework that measures the performance of AI agents across a wide variety of complex, real-world-inspired IT automation tasks.

It’s part of IBM Research’s Enterprise Agents and Benchmarks collection on Hugging Face, a family of open benchmarks for evaluating AI agents on enterprise tasks.

It measures agent performance across three IT personas:

• Site Reliability Engineering (SRE) — availability and resiliency
• Compliance and Security Operations (CISO) — compliance and security of IT implementations
• Financial Operations (FinOps) — cost efficiency and return optimization

Benchmarks already exist for pieces of this:

• SRE — AIOpsLab, InsightBench, TSB-AD, TrainTicket
• CISO — CIS Benchmark
• FinOps — nothing comparable; the FinOps Foundation publishes scorecards, but no benchmark that actually tests agent resolution

While these are valuable resources individually, they’re limited, they don’t reflect real-world IT problems well, they cover too narrow a slice of the IT landscape, and few of them automate evaluation end-to-end.

ITBench tries to address all three limitations at once, and the fact that FinOps essentially had no real benchmark until this paper is one of the more interesting gaps it fills

How ITBench is Built ?

• Scenario Specification and Environment
• AI Agent
• Leaderboard

1. Scenario Specification and Environment

A scenario in ITBench is formalized as a tuple p = ⟨M, E, T, D⟩:

• M (Scenario Specification) — metadata and deployment descriptors for the scenario
• E (Environment) — the operational testbed (e.g. a Kubernetes cluster, a RHEL9 host) where the problem actually occurs
• T (Triggering Events) — the alerts that fire when the fault manifests
• D (Desired Outcome) — the automation objective; what “solved” means

Each scenario also carries a ground truth — the task-specific expected outcome the Evaluator checks the agent’s final result against.

2. AI Agents

ITBench agents operate under a POMDP (Partially Observed Markov Decision Process) model:

• The agent never sees the full environment state directly, it only receives observations returned by tools
• It decides its next action based on the current observation plus the full history of past observations and actions.
• The environment itself evolves as a function of its previous state and the agent’s last action.
• In short the agent probes, decides, acts, and the loop repeats until it either reaches the goal state or gives up. Success is measured by comparing the final (terminated) state against the goal state.

ITBench’s baseline agents are built on different LLMs for both natural language reasoning and code generation. Each agent is initialized with a prompt describing its goal, context, task, and expected output format, and interacts with the environment in natural language through a toolbox.

3. Leaderboard

ITBench includes a leaderboard to promote reproducibility and comparative analysis, offering a predefined, extensible set of performance metrics.

For each scenario, ITBench records the final system state after task completion and validates it against ground truth at the end of the run.

Notably, it supports partial scoring, not just pass/fail, which gives more meaningful & constructive feedback than a binary result would.

How it Works ?

The end-to-end benchmarking flow, after an agent registers, looks like this:

1. New benchmark jobs go into a queue.
2. The Benchmark Runner pulls a scenario for a registered agent and provisions the environment per the scenario spec.
3. The environment is deployed and a fault is injected, the agent is only given access once the environment reports itself “Ready.”
4. The agent works the problem: it polls for readiness, then diagnoses and mitigates using its toolset, reporting STARTED and FINISHED status as it goes.
5. Once the agent finishes (or times out), the Benchmark Runner evaluates the final environment state against ground truth and tears the scenario down.
6. Results across all scenarios in the run are aggregated and published to the leaderboard.

The environment is fully under the agent’s control during this window, so it’s treated as a “playground” destructive actions are expected and accounted for.

The Results

1. pass@1 — used for both fault localization and mitigation.

• For fault localization, correctness means the predicted root cause exactly matches ground truth.
• For mitigation, correctness means the agent actually found, applied, and resolved the issue. Reported as a percentage.

1. Time-based metrics:

• Mean Time to Diagnose (MTTD) — time taken to successfully diagnose a scenario
• Mean Time to Repair (MTTR) — time taken to successfully execute a fix
• Time to Process (TTP) — time taken to determine compliance posture (pass/fail)

SRE:

• GPT-4o was the strongest performer: 13.81% pass@1 on diagnosis, 11.43% on mitigation — meaningfully higher than any other model tested.
• Llama-3.1–8B, despite far fewer parameters, had the fastest detection (57.50s MTTD) and repair (245.13s MTTR) among its successful runs.

CISO:

• GPT-based models dominate both pass@1 and TTP, pass@1 is nearly 2x the next-best models, while also processing scenarios fastest.

FinOps:

• GPT-4o led diagnosis at a 33% pass rate for identifying the cause of a cost increase.
• On cost and efficiency metrics specifically, performance was flat across all models, none achieved optimal CPU/memory cost or high CPU efficiency.

Four findings worth sitting with

1. Impact of scenario complexity

• SRE: Scenarios are categorized Easy/Medium/Hard based on fault propagation chain length, number of resolution steps, and technology diversity. Success rates decline clearly as complexity increases, GPT-4o (the best model) diagnosed only 36%, 7.73%, and 5.0% of Easy, Medium, and Hard scenarios respectively.
• CISO: Complexity maps directly to scenario class; Kyverno scenarios are Easy, k8s-opa and rhel-opa are Medium, Kyverno-update is Hard. All models degrade predictably as difficulty rises.
• FinOps: With only two scenarios (one Easy, one Hard), no model besides GPT-4o could even diagnose the hard scenario, and none could mitigate it.

2. Inherent non-determinism in the environment

GPT-4o stayed the top performer across all three personas, but it was far from deterministic, despite hyperparameter tuning aimed at consistency.

On one SRE scenario it diagnosed correctly in only 1 of 10 runs; on another, 8 of 10. Mitigation showed the same spread. This wasn’t unique to SRE — CISO and FinOps scenarios showed the same instability.

The paper attributes this to small real-time telemetry fluctuations altering the model’s token generation from run to run. Practically, this means a single “pass@1 score” for an agent isn’t really a fixed number, it’s closer to a distribution, and ITBench’s repeated-run methodology is what makes that visible in the first place.

3. Interdependence between diagnosis and mitigation

Diagnosis and mitigation are usually assumed to be tightly coupled: get the root cause right, and the fix should follow.

But the results don’t fully support that.

In one scenario, an SRE agent using llama-3.3–70b misidentified the root cause as a memory limit issue, when the actual cause was an HTTP request corruption fault; yet it still resolved the incident, essentially by accident, by scaling up service replicas as a generic remedy.

In another scenario labeled “Easy,” agents diagnosed the root cause with high confidence but couldn’t produce a working fix in any run.

So correct diagnosis doesn’t guarantee a fix, and a fix doesn’t require correct diagnosis. That’s a genuinely counterintuitive finding, and it says something about how brittle “generic” mitigation actions can be as a substitute for actually understanding the fault.

4. Impact of tracing on accuracy

ITBench’s alert-driven workflow mirrors how SREs are actually notified via golden-signal alerts from collected telemetry, not raw data dumps.

To test how much this telemetry actually matters, ITBench supports automated data masking, and the results are stark:

• Removing trace data dropped GPT-4o’s diagnosis pass@1 from 13.81% to 9.52%, and mitigation collapsed from 11.43% to 2.86%.
• One “Easy” scenario went from an 80% diagnosis rate with traces to 0% without them.

This is the finding that lines up most closely with what I’ve actually built.

Having spent the last year on the observability side, routing logs through Cribl, collecting metrics with Prometheus, visualizing through Grafana and Loki.

This result isn’t surprising from the infra side, but it’s rarely the headline when people talk about “AI agents for SRE.”

Everyone benchmarks the model. Almost no one benchmarks the instrumentation the model depends on.

And the paper notes only about 20% of real applications have tracing enabled at allb which means for most production systems today, agents are operating closer to the “no traces” number than the “with traces” one.

The ceiling on agent performance here isn’t model capability, it’s observability coverage. That’s a much less exciting thing to fix than “use a better model,” but it’s probably the more important one.

Takeaway

ITBench’s headline numbers, 13.8% resolved for SRE, 25.2% for CISO, 0% for FinOps, are a useful corrective against the current narrative that agentic AI is close to running IT operations unsupervised.

It isn’t, yet. But the more interesting part of this paper isn’t the leaderboard, it’s why agents fail:

• Thin observability
• Non-deterministic environments
• A diagnosis-mitigation gap

that no one’s really solved.

If you’re building in this space, that’s where the real work is.

Paper: ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks (Jha, Arora, Watanabe et al., IBM Research / UIUC, 2025)