Thousands of alerts, one root cause, and an engine allowed to say "I don't know"

Built: September 2026, for the Microsoft Codathon (Problem 3), team of five. Result: 1st in our college, 9,900 points. Status: shipped, open source.

When one microservice fails, every service that calls it starts failing too. The on-call engineer gets thousands of alerts, and the loudest one is usually a victim, not the cause.

Problem 3 at the Microsoft Codathon asked for an AI-powered root cause analysis engine. Our team, Ace Azael, built Antibody: it reads logs, metrics, traces and service events, folds everything that belongs together into one incident, and ranks the probable root causes with evidence and a confidence label. I led the technical explanation in our pitch.

The part I am proudest of is one rule we wrote down on day one.

Automation before AI

Every decision in Antibody is deterministic and replayable: detect, correlate, rank, suggest. The language model only rewrites evidence the engine has already computed into plain English, and that text is labelled as AI-generated. If the model is down, nothing breaks; a template writes the explanation from the same evidence.

That rule shaped everything else. Here is the pipeline:

ingest     learn the dependency graph from trace spans (caller -> callee)
detect     per-service EWMA baselines, z-scores, error-log and error-span ratios
           debounce: 2 bad ticks in a row before anything counts
correlate  a new abnormal service joins an open incident if it calls into the failure
rank       8 signals -> an RCA score from 0 to 1 (not a probability)
verdict    HIGH / MEDIUM / LOW from fixed rules. LOW means "insufficient evidence"
suggest    fault signature -> restart, roll back or scale out
human      acknowledge -> apply fix -> verify recovery -> resolve -> remember

The eight ranking signals include personalised PageRank on the anomaly graph, which suspect went wrong first, how many of the callers' error logs blame it, a local fault signature (crash, bad deploy, leak, CPU, slow queries), a recent deploy, and a match with a past resolved incident.

An engine that is allowed to say "I don't know"

Most demos rank something first and present it with total confidence. We wanted the confidence label to mean something, so it comes from fixed rules: how far the top suspect leads the second, how many independent kinds of evidence agree, and whether anything contradicts it. When the rules aren't satisfied, Antibody abstains.

The benchmark, including where it loses

84 scenarios: 7 services, 6 fault types, 2 seeds each, every one with a known cause. The engine never sees the ground truth; it is only used for scoring. python bench.py reproduces it in about 7 seconds.

Root cause ranked first	Clean	Hard mode (random red-herring blips)
Loudest anomaly only	88.1%	84.5%
Anomaly + earliest onset	100%	75.0%
Onset + dependency graph	100%	79.8%
Antibody	100%	84.5% (top 3: 92.9%)

On the confidence labels in hard mode: HIGH is given 78.6% of the time and is right 98.5% of the time. LOW, the abstain, is given 8.3% of the time, and a forced guess there would have been right only 42.9% of the time. When Antibody does answer, it is right 95.8% of the time.

Now the part a judge would find anyway, so we put it in the README first:

• On clean traffic, two simpler baselines also hit 100%. Our simulator is too easy to separate the models there.
• In hard mode the full model only ties "loudest anomaly" overall. It wins every quiet-crash cascade, where the root goes silent while its callers get loud. It loses when the fault is high in the graph and a red herring sits below it.
• The confidence label flags 5 of those 6 losses as LOW or MEDIUM. So when it is wrong, it mostly knows it is unsure.

Other numbers: two unrelated faults at once were found as separate incidents in 6 of 6 pairs, median time to detect is 2 seconds, and a database crash folds about 2,000 events into one incident with 5 services alerting and db ranked first.

What it doesn't do yet

• The 7 services are simulated. The ingest API accepts real data; we have not connected a real system.
• Fault signatures are pattern rules, so a new kind of failure gets "investigate", not a confident fix.
• Memory of past fixes is in-process and resets on restart.
• No auth on the API. It is a local demo.

The lesson I took from Antibody, and have reused in every build since: decide in code, explain with the model, and make "I'm not sure" a first-class answer.

Repo (MIT), with the benchmark and tests: github.com/victorisaacchinta-eng/antibody

---

I wrote this with AI assistance, working from the repo and its benchmark. Every number above comes from bench.py and the README.