10 Real Airflow Incidents, 5 AI Models — Which One Handled Production Best?

Suyash Magar — Sat, 26 Sep 2026 15:47:54 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

I built OpsBench - Airflow and SRE Troubleshooting, a benchmark designed to test whether AI models can diagnose realistic production issues rather than only answer coding questions.

The benchmark contains 10 tasks covering:

Airflow Slow DAG Parsing Diagnosis
Fixed EST vs Daylight Saving Time
Shell Script Failure Reported as Success
API Timeout and Retry Strategy
Airflow Logical Date vs Business Date
Airflow DAG Parser Scalability
Airflow Worker Deadlock Diagnosis
Batch Job Performance Regression
Distributed Concurrency Control
Noisy Production Incident Root Cause

I wanted to measure whether models could identify root causes, ignore misleading symptoms, reason about Airflow internals, and recommend production-safe fixes.

The tasks cover scenarios such as:

expensive top-level code slowing DAG parsing
DST-aware timezones vs fixed UTC-5 schedules
shell scripts returning non-zero exit codes
retry storms caused by unbounded API retries
business-date calculation mistakes
scheduler scalability issues
database deadlocks
batch jobs becoming 3x slower
concurrency across multiple workers
noisy production logs containing one real root cause

Models Tested

I evaluated the benchmark against five models:

Model	Score
Claude Sonnet 4.6	100
GPT-5.4 mini	100
GPT-5.5	100
Gemini 3.7 Flash	100
Qwen 3 Coder 480B	90

I intentionally selected models from different providers and capability/cost profiles so I could compare not only correctness, but also efficiency.

Findings

The biggest surprise was that four of the five models achieved a perfect score across all 10 tasks.

This suggests that current models are already quite capable at many common Apache Airflow and SRE troubleshooting scenarios when the problem contains enough evidence.

Another important finding was the relationship between cost and performance.

Some of the lower-cost models achieved the same perfect score as more expensive models. For this benchmark, higher inference cost did not automatically produce better results.

This changed the question for me from:

Which model is the most powerful?

to:

Which model is powerful enough for the task at the lowest practical cost?

That is especially relevant when building production AI assistants for incident triage or operational support.

The most interesting failure

Qwen 3 Coder 480B scored 90, failing only the Airflow DAG Parser Scalability task.

The model correctly identified that:

expensive operations were happening during DAG parsing
repeated parsing across 120 DAGs compounds the problem
increasing CPU, timeout, or parser processes does not fix the root cause
expensive work should be moved into Airflow tasks

However, it missed one operational consequence:

Repeated parse-time API calls and database queries can also put pressure on the external systems themselves, such as databases and APIs.

That difference was useful because it showed that a technically strong answer can still miss a broader systems-level impact.

Noisy production incident reasoning

One task intentionally included several distractions:

worker memory warning
DNS latency
DAG parsing delay
database CPU information
an actual database deadlock

The models had to prioritize direct evidence and identify the real issue rather than react to every warning.

The strongest evidence was the circular database lock wait combined with a recent change in transaction lock ordering.

The models generally handled this well, which suggests they can be effective at separating symptoms from root causes when the logs contain clear evidence.

What surprised me most

I expected the larger or more expensive models to consistently outperform smaller ones.

That did not happen in this benchmark.

Several models reached the same maximum score, while a lower-cost model could be significantly more cost-efficient.

So for this type of production troubleshooting, model selection may need to consider:

accuracy
latency
cost
operational reliability

rather than just choosing the largest available model.

What I would measure next

The current benchmark provides all relevant logs and context in a single prompt.

A future version would test interactive incident investigation.

For example, the model might initially receive only:

An Airflow task has been running for 45 minutes with no progress.

The model would then need to decide what evidence to request next:

task logs
worker CPU
database lock information
stack traces
scheduler health
network metrics

That would test whether the model knows how to investigate an incident, rather than simply recognizing the answer after all evidence has already been provided.

I would also like to test:

business holiday calendars
partial upstream data availability
retry multiplication across multiple layers
stale dependency state
multi-step incident response
tool-using SRE agents

My Benchmark

You can view the complete benchmark, tasks, model results, and leaderboard here:
OpsBench - Airflow and SRE Troubleshooting

AI Disclosure: AI assistance was used for article drafting and editing. The benchmark design, execution, results, and final review were performed by the author.

DEV Community: Suyash Magar