This is a submission for the Kaggle Benchmarking Challenge
What I Benchmarked
I built OpsBench - Airflow and SRE Troubleshooting, a benchmark designed to test whether AI models can diagnose realistic production issues rather than only answer coding questions.
The benchmark contains 10 tasks covering:
- Airflow Slow DAG Parsing Diagnosis
- Fixed EST vs Daylight Saving Time
- Shell Script Failure Reported as Success
- API Timeout and Retry Strategy
- Airflow Logical Date vs Business Date
- Airflow DAG Parser Scalability
- Airflow Worker Deadlock Diagnosis
- Batch Job Performance Regression
- Distributed Concurrency Control
- Noisy Production Incident Root Cause
I wanted to measure whether models could identify root causes, ignore misleading symptoms, reason about Airflow internals, and recommend production-safe fixes.
The tasks cover scenarios such as:
- expensive top-level code slowing DAG parsing
- DST-aware timezones vs fixed UTC-5 schedules
- shell scripts returning non-zero exit codes
- retry storms caused by unbounded API retries
- business-date calculation mistakes
- scheduler scalability issues
- database deadlocks
- batch jobs becoming 3x slower
- concurrency across multiple workers
- noisy production logs containing one real root cause
Models Tested
I evaluated the benchmark against five models:
| Model | Score |
|---|---|
| Claude Sonnet 4.6 | 100 |
| GPT-5.4 mini | 100 |
| GPT-5.5 | 100 |
| Gemini 3.7 Flash | 100 |
| Qwen 3 Coder 480B | 90 |
I intentionally selected models from different providers and capability/cost profiles so I could compare not only correctness, but also efficiency.
Findings
The biggest surprise was that four of the five models achieved a perfect score across all 10 tasks.
This suggests that current models are already quite capable at many common Apache Airflow and SRE troubleshooting scenarios when the problem contains enough evidence.
Another important finding was the relationship between cost and performance.
Some of the lower-cost models achieved the same perfect score as more expensive models. For this benchmark, higher inference cost did not automatically produce better results.
This changed the question for me from:
Which model is the most powerful?
to:
Which model is powerful enough for the task at the lowest practical cost?
That is especially relevant when building production AI assistants for incident triage or operational support.
The most interesting failure
Qwen 3 Coder 480B scored 90, failing only the Airflow DAG Parser Scalability task.
The model correctly identified that:
- expensive operations were happening during DAG parsing
- repeated parsing across 120 DAGs compounds the problem
- increasing CPU, timeout, or parser processes does not fix the root cause
- expensive work should be moved into Airflow tasks
However, it missed one operational consequence:
Repeated parse-time API calls and database queries can also put pressure on the external systems themselves, such as databases and APIs.
That difference was useful because it showed that a technically strong answer can still miss a broader systems-level impact.
Noisy production incident reasoning
One task intentionally included several distractions:
- worker memory warning
- DNS latency
- DAG parsing delay
- database CPU information
- an actual database deadlock
The models had to prioritize direct evidence and identify the real issue rather than react to every warning.
The strongest evidence was the circular database lock wait combined with a recent change in transaction lock ordering.
The models generally handled this well, which suggests they can be effective at separating symptoms from root causes when the logs contain clear evidence.
What surprised me most
I expected the larger or more expensive models to consistently outperform smaller ones.
That did not happen in this benchmark.
Several models reached the same maximum score, while a lower-cost model could be significantly more cost-efficient.
So for this type of production troubleshooting, model selection may need to consider:
- accuracy
- latency
- cost
- operational reliability
rather than just choosing the largest available model.
What I would measure next
The current benchmark provides all relevant logs and context in a single prompt.
A future version would test interactive incident investigation.
For example, the model might initially receive only:
An Airflow task has been running for 45 minutes with no progress.
The model would then need to decide what evidence to request next:
- task logs
- worker CPU
- database lock information
- stack traces
- scheduler health
- network metrics
That would test whether the model knows how to investigate an incident, rather than simply recognizing the answer after all evidence has already been provided.
I would also like to test:
- business holiday calendars
- partial upstream data availability
- retry multiplication across multiple layers
- stale dependency state
- multi-step incident response
- tool-using SRE agents
My Benchmark
You can view the complete benchmark, tasks, model results, and leaderboard here:
OpsBench - Airflow and SRE Troubleshooting
AI Disclosure: AI assistance was used for article drafting and editing. The benchmark design, execution, results, and final review were performed by the author.
Top comments (1)
Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support