**
๐ Hypocenter: The On-Call Agent That Wonโt Blame Your Deploy Without Proof
**No path, no blame. How a graph made my AI agent honest ๐
๐จ 13:40. Production is on fire.
30% of mobile logins are failing. Seven services are screaming red. ๐ด Three deploys, a config change and a feature flag all landed in the last hour.
Which one broke production? ๐ค
Most incident tools just show you that things are burning. Some take a guess at why, and the guess is almost always โthe latest deploy.โ On the faults I planted for testing, that guess is right only 50% of the time. ๐ฌ The other half? You roll back the wrong thing while the real cause keeps burning. ๐ฅ
๐ก The Big Idea
I built Hypocenter for Graph Hacks (WeMakeDevs ร FalkorDB) around one unbreakable rule:
โก No path, no blame. A change can be a suspect only if the graph connects it to the failing service, and it leads only if the alerts climb that path in order.
๐ Fun fact: an earthquakeโs epicenter is where you feel it on the surface. The hypocenter is where it actually started, deep underneath. Thatโs exactly what this agent hunts for. ๐ฏ
๐ง How It Works
Hypocenter keeps a FalkorDB graph for every company ๐ธ๏ธ: services and who calls whom, owners and the on-call rota, every deploy, config change and flag, every alert, and every incident already solved.
When an incident opens, four agents take it in turn: ๐ต๏ธ Triage โ ๐ฌ Investigator โ ๐คจ Skeptic โ ๐ ๏ธ Responder.
The twist? They never talk to each other directly. Each writes its handoff into the graph, and the next one reads it back. ๐
Hereโs the login meltdown, solved:
๐ A deploy hit auth at 13:34
๐ด auth started alerting at 13:38
๐ฑ The mobile backend above it at 13:40: the alerts climbed the path in order โ
๐ฉ A feature flag on users sits on the same path, but users never alerted. Rejected. โ
๐งฉ A graph algorithm splits the rest of the alerts into three separate problems and links the other two to the incidents they belong to
๐ก๏ธ The Evidence Gate: Proven, or It Tells You Why Not
Before any verdict reaches you, it passes 8 named checks, each shown with the exact query that decided it:
๐ Is the suspect actually in the graph?
๐ค๏ธ Is it on a call path?
โฑ๏ธ Did it fail first?
๐ Do the alerts climb in order?
๐งพ Does it explain every alert?
๐ฅ Does every rival fail a check?
โ
Is every claim backed by the graph?
๐ Did anything fail off the graph?
All eight pass โ CONFIRMED ๐. If the graph canโt connect the failure, it says so instead of guessing. Because no path is not proof of safety. ๐
๐งช It Proves the Fix Before You Click Approve
The agent can only propose. A human always approves. ๐ค But first, Hypocenter applies the fix to a copy of the graph and tells you exactly which alerts it will clear and which it will leave. ๐ฎ
After approval, it checks that the alerts really cleared. If they didnโt? The incident reopens on its own. ๐
๐ But Does It Actually Work?
I tested it on all 360 real failures in the public RCAEval benchmark, against the benchmarkโs own baseline code:
๐ฏ 81%: right service first
๐ 71% on 90 failures I kept locked away while building it. A published method, NSigma, scores 76% there, and I report that loss instead of hiding it. ๐ฏ
๐ชค 94.5โ96.8% on planted faults packed with decoys, vs 50% for โblame the latest deployโ
And to prove the graph does the real work: a test for each of the 8 FalkorDB algorithms changes what that algorithm reads and checks that the decision flips. ๐ช Pull FalkorDB out entirely? Thereโs no answer at all.
๐ข It Knows Your Company Too
๐ฌ Ask โwhy was this change made?โ and the answer comes straight from the teamโs chat
๐ง Ask โwho owns it?โ and it remembers what โitโ meant
๐ฆ Before a deploy, it says ship, review or hold, with every reason
๐ Try It Now
๐ Live demo: 16-4-94-72.sslip.io (no sign-in, no key)
๐ป Code: github.com/rago12bit/grapho1, running with one Docker command
โญ If your incident tool has ever confidently blamed the wrong deploy, give it a path to walk.
Top comments (0)