Blast Radius: many alerts, one cause — building AI on-call agents on a graph

Ahmedulla — Sun, 18 Oct 2026 05:50:39 +0000

It's 2 AM. Your phone goes off. Checkout is failing. So is the frontend, the payment service, the proxy in front of
everything, and a couple of services you've never touched. A wall of alerts, and only one of them is the actual problem.

Before you can fix anything, you have to answer three questions:

Which service is the cause, and which ones are just victims?
What changed recently near it?
Who should be woken up?

None of those answers sits in one place. They're in the connections. Service A calls B, B reads from database C, C
was changed by a pull request two hours ago, the author of that PR is on the team that owns C, and this week that
team's on-call is someone else entirely. That's a graph problem. So I built the agents on a graph.

Blast Radius is a small team of AI on-call agents that use FalkorDB as their
context, their memory and their audit trail. I built it for Graph Hacks 2026 (WeMakeDevs × FalkorDB). This post
walks through the graph model, the queries doing the real work, how the agents remember past incidents, and how I
kept the whole thing running on free-tier LLMs.

Code: https://github.com/AgentAhmed/blast-radius

Agent questions are graph questions

Here's what the investigator agent needs to know during an incident, and the shape of the query behind each:

Question	Graph shape
What breaks if `payment` fails?	variable-length path, walked backwards along dependencies
What could cause all of these alerts?	variable-length path forward from every alerting service, counted
What changed near it?	one hop from component to `PullRequest`, filtered by time
Who's on call for it?	component → team → on-call rotation → person
Have we seen this before?	vector search over past incidents, re-ranked by graph structure

In SQL, most of these are recursive CTEs or several round trips. In Cypher, each one is a few lines. That matters
twice: an LLM can call it as a tool, and a human can read it when the agent shows its working.

The graph model

Each organisation gets its own graph (org_<tenant>) in the same FalkorDB instance, so memory never leaks between
companies.

Topology: Component nodes (services, datastores, Kafka topics) with CALLS, READS_WRITES, PUBLISHES and CONSUMES edges, plus SERVES edges to the user flows they power (checkout, browse, cart).
People and ownership: Person -MEMBER_OF-> Team -OWNS-> Component, and on-call shifts.
Change: PullRequest -MODIFIES-> Component, Person -AUTHORED-> PullRequest.
Knowledge: design decisions (ADRs), runbooks and documents, linked to the components they describe.
Memory: Incident, Alert, Action and Lesson, with ROOT_CAUSE, IMPACTED and RESOLVED_BY edges.
The agents themselves: AgentRun and Step nodes for every LLM call and tool call, Task nodes for hand-offs, and SUPPORTED_BY edges from a finding to the exact things it cites.

Three modelling decisions mattered more than the rest.

Agent state lives in the same graph as the data. A step that ran a query has TOUCHED edges to the nodes it
read. "Show me every incident where the agent looked at this PR" is just a query, not a log search.

Evidence is edges, not text. A finding doesn't only say a PR caused it. It has a SUPPORTED_BY edge to that
PR. The UI can draw it, and the agent can't cite something that doesn't exist.

Timestamps are stored as numbers but shown to the LLM as "2h ago". Small models are surprisingly bad at epoch
arithmetic. This one change fixed several wrong answers in testing.

The root-cause query

This is the heart of the project. Given the alerting components, find what they have in common downstream:

UNWIND $alerting AS alerting_name
MATCH (a:Component {name: alerting_name})-[:CALLS|READS_WRITES|PUBLISHES|CONSUMES*0..6]->(c:Component)
WITH c, count(DISTINCT a) AS covers
OPTIONAL MATCH (c)-[:CALLS|READS_WRITES|PUBLISHES|CONSUMES]->(d:Component)
WHERE d.name IN $alerting
WITH c, covers, count(DISTINCT d) AS alerting_deps
OPTIONAL MATCH (c)<-[:MODIFIES]-(pr:PullRequest)
WHERE pr.merged_at >= $since
WITH c, covers, alerting_deps, count(DISTINCT pr) AS recent_changes
RETURN c.name AS component, covers, c.name IN $alerting AS is_alerting, alerting_deps, recent_changes
ORDER BY covers DESC, is_alerting DESC, alerting_deps ASC, recent_changes DESC, component
LIMIT $limit

The ordering reads the way an SRE thinks:

covers: how many alerting services depend on it. The real cause should explain all the symptoms.
is_alerting: it's showing symptoms itself.
alerting_deps (fewer is better): if its own dependencies are alerting, it's probably a victim too.
recent_changes: something merged nearby recently is suspicious.

The *0..6 is deliberate. Zero hops means an alerting service can be its own root cause.

My favourite test case is the OpenTelemetry Demo's kafkaQueueProblems fault. The services that alert,
accounting and fraud-detection, don't call each other. The only thing they share is the Kafka topic orders,
and the topic never alerts. Searching logs would never point at it. The query does, because both consumers
depend on it.

Don't let the LLM ignore the graph

The LLM (a free-tier open model) decides which tools to call and writes the final finding. Early on, it would
sometimes run the right query, get the right answer back, and then submit something else. Usually the loudest
alerting service.

So now the graph checks the answer. When the investigator submits a root cause, the coverage check runs again:

If another component explains more of the alerts, the answer goes back once, with numbers: "only 2 of 4 alerting components depend on checkout, while payment is a dependency of 4".
After one rejection, the next answer is accepted. The graph gives a second opinion, it doesn't get a veto. The model might know something the graph doesn't.

That single check fixed every miss in my evaluation.

Memory that makes the second incident faster

When an incident is resolved, a scribe agent writes a postmortem and a lesson, and links them to the root cause and
the fix. Next time something similar happens, the investigator calls the similar-incidents tool, which mixes vector
search with graph structure:

CALL db.idx.vector.queryNodes('Incident', 'embedding', $k, vecf32($vec)) YIELD node, score
WITH node AS i, 1 - score AS text_sim
WHERE i.id <> $exclude AND i.status = 'resolved'
OPTIONAL MATCH (i)-[:ROOT_CAUSE]->(rc:Component)
...
RETURN i.id, rc.name, fixes,
       0.6 * text_sim + 0.3 * same_root + 0.1 * overlap AS score

Text similarity alone mixes up "payment errors" and "payment latency". Adding "same root-cause component" and
"overlapping impacted services" from the graph makes the match a lot sharper. One FalkorDB detail that cost me
some time: queryNodes returns cosine distance, so similarity is 1 - score.

The embeddings are local (bge-small-en-v1.5 through fastembed). Recall costs nothing and never hits a rate limit.

Agents that hand off work through the graph

There are four agents:

Triage groups alerts into an incident and sets the severity. No LLM: it's pure graph logic, so it's instant and free.
Investigator finds the cause and acts on it.
Scribe writes memory after a human resolves the incident.
Company brain answers questions like "why does checkout publish to Kafka, and who decided that?"

They never call each other directly. They coordinate through Task nodes:

Triage creates an investigate task.
A worker claims it with one Cypher query that grabs the oldest open task and marks it claimed. FalkorDB applies writes to a graph one at a time, so two workers can't claim the same task.
When a human resolves the incident, a record task hands it to the scribe.

If a worker dies, the task is still sitting in the graph, and anyone can see who is working on what.

Surviving free-tier rate limits

I wanted this to run on free LLM tiers, which means rate limits aren't an edge case, they're normal. Two things make
it work.

A fallback chain. Groq first, then any other configured provider (Gemini, OpenRouter, OpenAI, Claude, or any
OpenAI-compatible endpoint), then a local Ollama model. Errors are classified (rate limited, bad request, bad key,
model not found, provider down), and each model gets a cooldown instead of being hammered in a retry loop.

Checkpoints in the graph. Every LLM response and every tool result is a Step node, chained with NEXT. When
every provider is exhausted, the run is marked paused. Later, the worker rebuilds the conversation from the stored
steps and carries on exactly where it stopped. Nothing is re-queried and nothing is lost.

One bug I only found by testing on the real free tier: after a long pause, a resumed run sometimes kept calling
query tools until it ran out of budget, and finished with no answer at all. The fix was small. Once the budget is
reached, the model is offered only the final-answer tool. It can't wander off any more, so it summarises what it
already found.

Does it work?

I wrote an evaluation that injects each of the OpenTelemetry Demo's 8 fault flags into a fresh graph, runs the real
investigator, and scores it against hidden ground truth. Then it resolves the incident and injects the same fault
again to test memory.

	First occurrence	Same fault again
Root cause correct	8/8	8/8
Cited the incident it had just learned from	—	7/8
Avg LLM calls / graph queries	5.6 / 4.5	5.0 / 4.8

During that run, the free tier forced 24 model fallbacks and 34 full pauses. Every paused run resumed from its
checkpoint and finished.

It's an honest but small test: 8 simulated faults on a realistic topology. Real incidents are messier, with partial
outages and several things breaking at once.

Real data, not just a demo

A demo dataset only proves so much, so there are two ways in for real data:

GitHub: point it at any public repo. Services and dependencies come from its compose files (including hostnames in environment variables like http://payment:8080), owners from CODEOWNERS, and changes and experts from merged PRs, mapped to components by file path. On the OpenTelemetry Demo repo it found 26 components and 43 dependencies. GitHub logins are stored as stable pseudonyms by default.
Alerts: an Alertmanager-compatible webhook, which Grafana can post to as well. Labels like service, app or a pod name (checkout-5c9d7f-q2w8r) are matched to components. Anything that doesn't match is reported back, never guessed.

And the agents act in real tools:

A GitHub issue with the root cause, the evidence and the recommended fix. The token is fine-grained: one repository, Issues permission only.
A Slack or Discord message when the cause is identified (severity, root cause, confidence, who was paged, the issue link), and another when it's resolved.

Every action has a fixed id per incident, so a retried or resumed run never pages twice or opens a duplicate issue.

What I learned

Give the LLM queries, not the database.I gave the model sixteen named, tested graph tools instead of letting it write Cypher. Every query is one I can test and explain, the UI shows the exact query behind each step, and a model can't run a query I didn't write.
Let the graph check the model, not just feed it.
Keep agent state next to the data. Checkpoints, hand-offs and evidence as nodes and edges gave me resume after a crash, provenance and an audit trail almost for free.

What's next

Building dependencies from live OpenTelemetry traces instead of compose files, using resolved incidents to tune the
ranking weights, and letting the agent run approved fixes (a feature-flag flip or a rollback) behind a human
approval step.

Try it

It runs locally with Docker and a free Groq key, or Ollama with no key at all:

git clone https://github.com/AgentAhmed/blast-radius && cd blast-radius
docker compose up -d
# then follow the Quick start in the README

Inject paymentFailure, watch the investigator trace it, resolve it, then inject it again and watch it remember.

Here's the full demo:

The code, the Cypher queries, a design document and the evaluation are all on GitHub:
https://github.com/AgentAhmed/blast-radius

DEV Community: Ahmedulla