Can AI Diagnose a Linux Production Incident Without Making It Worse?

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

I wanted to test a practical question:

Can an AI model diagnose a Linux production incident from evidence, rather than simply recognize a familiar error message?

For my benchmark, I created a Linux SRE Incident Diagnosis task focused on filesystem exhaustion and application failure.

The incident involved a production server where an application began returning HTTP 500 errors.

At first glance, the evidence looked straightforward:

/var = 97% full

The application was also reporting:

errno=28 No space left on device

But the incident contained a deeper Linux troubleshooting clue.

The diagnostic evidence included:

lsof +L1

api 18472 app ... 47185920000 ... /var/log/api.log (deleted)

In other words, the api process was still holding an approximately 47 GB log file open even though the file had been deleted.

That creates an important distinction between what df reports at the filesystem level and what du can see through the visible directory structure.

I designed the benchmark to test whether the model could connect those pieces of evidence rather than stopping at:

"The disk is full."

The task evaluates six areas:

1. Root Cause — identify what actually caused the incident.
2. Evidence — cite the relevant Linux and application evidence.
3. Diagnostic Reasoning — connect the evidence logically.
4. Recommended Diagnostic Commands — suggest commands that would confirm the diagnosis.
5. Safe Remediation — recommend controlled recovery rather than destructive actions.
6. Risk Assessment — explain operational risks using the available evidence.

I chose this problem because it represents the kind of incident where a technically plausible answer can still be operationally dangerous.

Models Tested

For the first version of the benchmark, I tested:

Gemini 3.7 Flash

I intentionally started with one model.

My first goal was to validate the benchmark itself: could the task distinguish a shallow "disk full" answer from a response that actually connected filesystem usage, application errors, process-level evidence, remediation, and operational risk?

The published Kaggle benchmark currently contains 1 task and 1 evaluated model. Gemini 3.7 Flash received a 100.00 score, and the associated task is marked Pass.

Rather than immediately adding several models and producing a larger but less focused leaderboard, I wanted to establish a clear baseline first.

The next useful experiment would be to run the same task against additional models and compare their failure patterns, not just their final scores.

Findings

The headline result is:

Gemini 3.7 Flash — 100.00

But the number itself is not the most interesting part.

What interested me was the reasoning the benchmark was designed to test.

1. "Disk full" is not the complete diagnosis

A model can easily notice:

/var = 97%

and:

errno=28 No space left on device

But that is only the beginning.

The stronger diagnostic question is:

Why is the filesystem full, and why might the visible files not account for all of the space?

That is where the deleted-but-open log file becomes important.

2. df and du tell different stories

This incident demonstrates why Linux troubleshooting sometimes requires comparing different evidence sources.

df reports filesystem-level space consumption.

du reports disk usage associated with visible files and directories.

If a process still has a deleted file open, the storage can remain allocated even though the filename has disappeared from the directory tree.

That means a mismatch between filesystem usage and visible file usage is itself a diagnostic clue.

3. Process-level evidence matters

The benchmark explicitly requires the model to recognize the significance of:

lsof +L1

This command can expose deleted files that are still open by processes.

In this scenario, that evidence connects the filesystem problem directly to the api process and the deleted api.log.

That is much stronger than simply saying "check the logs."

4. Correct diagnosis is not enough

I also wanted the benchmark to test operational judgment.

A model might correctly identify that /var is full but then recommend indiscriminately deleting log files, rebooting the server, or killing the application process.

Those actions can have consequences in a production environment.

A safer response should first establish what is consuming the space and then choose a controlled remediation appropriate to the process and service.

That distinction is important to me because an AI assistant used in an SRE workflow should not only answer:

"What is wrong?"

It should also reason about:

"What is safe to do next?"

What surprised me?

The biggest lesson from building the benchmark was how much the evaluation design matters.

If I only checked whether the model mentioned "disk full," the benchmark would be easy to game with a shallow answer.

Instead, I had to make the evaluation evidence-specific:

• /var must be connected to the incident;
• errno=28 must be recognized;
• the deleted api.log must be identified;
• the api process must be connected to the open file;
• lsof +L1 must appear as an appropriate diagnostic command;
• remediation must account for operational risk.

That changed how I think about AI evaluation.

A benchmark should not only ask:

Did the model produce the expected answer?

It should ask:

Did the model demonstrate the reasoning that makes the answer trustworthy?

What would I measure next?

The obvious next experiment is to run the same benchmark against additional models.

I would particularly want to compare:

• whether models recognize deleted-but-open files;
• whether they understand the df versus du distinction;
• whether they recommend evidence-gathering before remediation;
• whether they suggest disruptive actions too early;
• whether their risk assessments are grounded in the incident evidence.

I would also like to expand the benchmark into additional Linux/SRE incident types while keeping the same evidence → diagnosis → remediation → risk structure.

That would make it possible to study failure patterns, rather than treating a single aggregate score as the whole story.

My Benchmark

You can inspect the public benchmark, task, results, description, and provenance on Kaggle:

Linux SRE Incident Diagnosis — Kaggle Benchmark

The benchmark is released under the Apache 2.0 license.

The current public leaderboard shows:

Gemini 3.7 Flash — 100.00

The benchmark contains one Linux SRE incident diagnosis task and is intended to evaluate evidence-based troubleshooting, diagnostic reasoning, safe remediation, and operational risk assessment.

Building this benchmark gave me a different way to look at AI-assisted infrastructure work.

For me, the interesting question is no longer simply:

"Can an AI find the answer?"

It is:

"Can an AI show enough evidence-based reasoning to help an engineer make a safe decision?"

That is what I want to measure next.

kagglechallenge #ai #machinelearning #linux #sre #devops