Seven Alerts at 2 AM: Building an AI On-Call Agent with a Graph Database

Akshay Shah — Sun, 18 Oct 2026 12:26:36 +0000

It's 2 AM. Your phone buzzes, then buzzes again. Seven alerts in four minutes.

Frontend says Orders failing. The API gateway is throwing 5xx errors. Checkout is erroring, payment is slow, cart is slow. Somewhere near the bottom there's a quiet warning on the cache, and a High CPU alert on the recommendation service.

Which one do you look at first?

Most people start with the loudest one. That's usually wrong. The loud alerts are symptoms. The real cause is often a small change on a shared service that everything else depends on. Finding it means connecting a lot of dots: which service calls which, who deployed what and when, who owns it, and what fixed this last time.

That's what I built for Graph Hacks (WeMakeDevs × FalkorDB): RootCause Agent, an AI on-call agent that finds the real cause of an outage by walking a graph.

🔗 Live app: https://rootcauseagent.streamlit.app
💻 Code: https://github.com/beyondions/rootcause-agent
🎥 Demo: https://www.youtube.com/watch?v=YDL6ZGoFmOY

Where I started

Four days ago I had never written a line of Cypher. I didn't know how an AI agent calls tools. I'd heard "graph database" but couldn't have told you why you'd pick one over Postgres.

I chose Track 1, Agents That Act on Connected Data, because on-call incidents felt like the clearest case where the answer lives in the connections, not in any single document.

Why this is a graph problem

Think about what an on-call engineer actually needs to know:

Is this alert the cause or a symptom? That depends on what the service depends on.
What else will break? That's everything that depends on the failing service, several hops deep.
What changed? A deploy, connected to a service, connected to the engineer who made it.
Who do I wake up, and how was it fixed last time? Owners, past incidents, the people who resolved them, and the runbook they used.

Every one of those is a path through connected data. So I modelled it as a graph in FalkorDB: 25 services of an online shop, plus teams, engineers, deploys, alerts, past incidents and runbooks, joined by relationships like DEPENDS_ON, OWNS, AUTHORED, DEPLOYED_TO, FIRED_ON and RESOLVED. All the data is synthetic.

How the agent finds the root cause

The rule turned out to be simple once I saw it as a graph:

An alerting service is a symptom if something it depends on is also alerting. What's left are the candidates, ranked by how many alerting services depend on them, and then by whether they had a recent deploy.

In my scenario, frontend, API gateway, checkout, payment and cart are all symptoms, because each of them sits upstream of another alerting service. Two candidates are left:

cache explains 5 alerts and had a deploy 11 minutes before the outage
recommendation explains only 2, and had no recent deploy

So the answer is cache, and the deploy behind it is D-1: "Config change: maxmemory lowered". The recommendation CPU alert is a red herring.

The core of it is one line of Cypher, a variable-length traversal:

MATCH (:Service {name: $service})-[:DEPENDS_ON*1..]->(dep)
RETURN DISTINCT dep.name

*1.. means "follow DEPENDS_ON one or more hops". In SQL that would be recursive joins. In Cypher it's two characters.

From there, the agent also finds the evidence path (frontend → API gateway → cart → cache), the blast radius (5 critical services), who to wake up (Priya, who fixed the same problem before, and Arjun, who made the change), the fix from the past incident's runbook, and then it acts: it writes a new incident back into the graph.

My design rule: facts from the graph, the LLM only explains

The first versions of the agent let the LLM write the whole answer. It was fluent and confident, and sometimes wrong.

The moment that changed my approach: the agent suggested a fix called "Restore Cache Memory Settings". That runbook doesn't exist. The real one is "Roll back cache config". My tests didn't catch it, because they only checked the root cause.

At 2 AM, a made-up runbook name is worse than no answer. So I set a rule:

Facts come from the graph. Structure is enforced in code. The LLM only explains.

The LLM still decides which graph tools to call and writes the explanation. But the lines that matter (the first line, the suggested fix, yes/no answers like "is recommendation the cause?") are rebuilt in code from the tool results before anyone sees them. [Adjust if needed: the contact line got the same treatment after the model once invented a team name.]

Proving the graph actually does the work

It's easy to say a graph helps. I wanted to measure it.

So I built a vector-RAG baseline over the same data: every node and relationship turned into a sentence, TF-IDF retrieval with stopwords and stemming, the current alerts added to the query, top 8 documents, same LLM. I tried hard to make it fair. My first version was accidentally unfair, and when I fixed it, RAG started finding cache too. That changed my pitch from "RAG gets it wrong" to "here's what RAG can't do".

Then I scored both against an answer key read from the graph, not typed in by hand:

	Graph agent	Vector RAG
Score (11 checks)	11/11	1/11

RAG could guess that the cache was involved. But it blamed the API gateway as the primary driver, it never found deploy D-1 or who made it, it couldn't list the full blast radius, it couldn't say who to wake up and why, and it can't take action. Asked "what changed?", it answered an increase in cache latency, which is an alert, not a change.

Same data, same model. The only difference is the graph.

Mistakes I made along the way

My first root-cause query returned all zeros. I tried to do everything in one big Cypher query. I split it into simple traversals and combined the results in Python, and it worked straight away.

shortestPath inside MATCH threw an error. In FalkorDB it has to go in RETURN or WITH. That took me a while to figure out.

My blast radius was hard-coded in the tests. I noticed when I asked myself why it never changed. Now every expected value comes from the graph.

The agent treated every question as the current outage. I typed "auth is not working, who is responsible?" and it blamed the cache, woke up the wrong people, and opened an incident. Auth wasn't even alerting. That led to service-check mode: if you ask about a service with no alerts, the agent checks its dependencies in the graph, flags a possible monitoring gap, names the owners, and opens nothing. It uses zero LLM tokens.

I ran out of tokens. Groq's free tier gives about 200k tokens per model per day, and a full investigation costs about 11k. Two test runs used up a whole day. I added a model fallback chain, an answer cache, and token logging, so the demo keeps working.

Deployment had its own surprises. My .env still pointed at localhost while I was trying to reach FalkorDB Cloud, and I once pasted the port into the host field, so it tried to connect to ...cloud:59676:59676.

Try it

🔗 Live app: (https://rootcauseagent.streamlit.app)
💻 Code: https://github.com/beyondions/rootcause-agent
🎥 Demo: https://www.youtube.com/watch?v=YDL6ZGoFmOY

Try "Orders are failing. What's the root cause and who do I wake up?", then "Is postgres healthy?"

What's next

Right now it's one scripted outage on synthetic data with a fixed clock. The real next step is feeding it a real service map (for example from OpenTelemetry) and real deploy history, then seeing whether the same rule holds up.

What I learned

Four days ago I didn't know Cypher. Now I understand why some questions are graph questions, and why an agent that answers at 2 AM should take its facts from data, not from the model's imagination.

Thanks to WeMakeDevs and FalkorDB for the push.

FalkorDB #GraphHacks #WeMakeDevs #AIAgents #GraphDatabase #SRE

DEV Community: Akshay Shah