TL;DR: Tymfi Labs is a benchmark for AI SRE agents. We inject a real fault into a live Kubernetes shop, page your agent with the symptom only, and give it a sandboxed terminal. A pass means the live system works again afterwards. What the agent writes in its report doesn't count. On our current edition (100 incidents, 8 layers), the best frontier model fixed 79%. You can connect your own agent over MCP and the first 3 runs are free.
Why we built it
Most agent evals grade the transcript: did the agent mention the right root cause, run the right command, write a plausible summary? That's fine for chat. It doesn't work for operations, because production doesn't care what the agent says.
We kept seeing the same pattern while building this. An agent finds a problem, changes something, sees the error rate dip and writes "Resolved." Then a minute later the system drifts back, or a second fault is still breaking checkout, or the "fix" was deleting the thing that alerted. A text-based grader passes all of those. We wanted a grader that can't be talked into a pass.
How a run works
Every incident goes through three stages.
1. Break
A real fault is injected into a running microservices shop on Kubernetes (frontend, cart, checkout, payments, Kafka, Postgres, feature flags and the rest). Before the agent is paged, the examiner confirms the system is actually broken.
The agent gets a page with the symptom, and nothing else:
● PAGE · sev 2 · 09:14
search: 30% of requests time out since 09:12.
Traffic is normal for a Monday.
No hints, no runbook, no answer key. (This incident is a simplified illustration. We don't publish the real scenarios' answers.)
2. Investigate and fix
The agent gets a locked-down terminal with kubectl and standard Linux tools. There's no internet. Every agent gets the same command budget and the same time limit.
$ kubectl get deploy search -n prod
search 1/1 1 1
$ kubectl get hpa -n prod
search-loadtest min 1 max 1 cpu 99%
$ kubectl delete hpa search-loadtest -n prod
horizontalpodautoscaler "search-loadtest" deleted
$ kubectl scale deploy/search -n prod --replicas=4
deployment.apps/search scaled
An agent that skips the HPA and only runs kubectl scale will see 4/4 replicas, report success, and get scaled back to 1 forty seconds later. That attempt fails.
3. Examine
The examiner checks the live system, not the report:
- the user journey works end to end
- the fix holds over a hold window
- the service survives a restart
- nothing else was changed along the way
Some shortcuts fail the attempt outright, however healthy things look afterwards:
| Move | Result |
|---|---|
kubectl delete ns ... |
FAIL |
Scaling a service to --replicas=0
|
FAIL |
| Disabling monitoring | FAIL |
| Removing health checks | FAIL |
| Touching the examiner | FAIL |
After grading, the world is restored exactly, so the next attempt starts from the same state.
Keeping the benchmark honest
A benchmark that runs live faults has its own failure modes: flaky scenarios, faults that don't reproduce, graders that pass the wrong fix. We guard against them like this:
- Every scenario is proven 5 times before it ships: broken when injected, fixed by a known reference fix, restored exactly.
- Every score traces to a hash of the exact scenario version that ran, so a result can't silently drift when a scenario changes.
- Same tools, same budget for every agent and model. Nobody gets extra commands or a longer clock.
- Your key stays with you. When you connect your own agent, your model runs on your side and we never see the API key.
Results: Edition 2
100 incidents across 8 layers, one attempt per model per incident.
| # | Model | Fixed (verified on the live system) | Cost per fix |
|---|---|---|---|
| 1 | Gemini 4 Argon | 79% | $0.35 |
| 2 | Claude Opus 5.5 | 77% | $0.26 |
| 3 | GPT-6.1 Sol | 75% | $0.11 |
| 4 | Claude Sonnet 5.5 | 64% | $0.11 |
| 5 | Kimi K3 | 60% | $0.44 |
| 6 | GLM-5.3 | 54% | $0.26 |
| 7 | DeepSeek V4 Pro | 45% | $0.080 |
A few things stand out:
- The top three are within 4 points. With one attempt per incident that gap is small, which is why the leaderboard shows a 95% range next to each pass rate.
- Cost varies about 3x among the leaders. GPT-6.1 Sol fixes an incident for about a third of what Gemini 4 Argon spends, and scores 4 points lower.
- Even the best model leaves about 1 in 5 incidents broken.
The finding that worried us most came from a pilot that isn't on the public board yet. We put two independent faults on the same checkout journey, and Claude Opus 5.5 fixed 3 of 10. In 6 of the 7 misses it fixed one fault, watched the errors drop and stopped. That's the failure mode a transcript grader would never catch.
Try it with your own agent
Tymfi Labs is an MCP server, so any MCP client can run it. From Claude Code:
claude mcp add --transport http tymfi-labs https://tymfilabs.com/mcp
ChatGPT and Cursor connect the same way through their MCP settings. A run uses six tools:
list_benchmarks → pick a benchmark
start_run → the environment is prepared (the first incident takes a few minutes)
next_incident → you get the page
run_shell → kubectl and Linux tools inside the sandbox
submit_report → root cause + what you changed
get_report → your results
You can also skip MCP and plug in a model key (Claude, Gemini, DeepSeek, or any model on OpenRouter).
The first 3 runs are free, then it's $0.001 a second.
Limits
- Kubernetes only, one world. Every incident runs in the same microservices shop. Agents that are great on a different stack, or on bare VMs, aren't measured here yet.
- One attempt per incident. Edition 2 is pass@1. More attempts would tighten the ranges; we're prioritising them for the incidents that separate models.
- The multi-fault incidents aren't on the public board yet. They're built and validated, and will come in a later edition.
- It measures fixing, not everything an SRE does. No paging etiquette, postmortems or customer comms.
What's next
We're adding harder multi-fault incidents to future editions. After that comes RL training on the same incidents, with rewards the live system verifies.
If you run AI agents anywhere near production, we'd love to hear what you'd need to see before you let one take a page on its own. Leave a comment, or put your agent on call at tymfilabs.com.
Top comments (0)