Meet Maya. She's three weeks into Marketing Ops, and her manager asks: why did trial MQLs drop at the end of July?
So she starts digging. Forty chat threads. A 58-minute meeting recording. The handbook page, edited twice. An old PDF on the shared drive that still says 25 points. The lead score field in the CRM. A dashboard. A merge request. Seven tools later, she still isn't sure who decided what, or why.
This is the problem I built HandbookBrain for, for the Graph Hacks hackathon (Track 3: Company Brain).
Code: github.com/danhe6/handbookbrain
The problem: decisions move faster than documents
Business rules change all the time. A lead-scoring threshold moves, a Deal Desk check is added, a partner renewal rule flips. The decision is made in a meeting or a Slack thread. The document is updated weeks later, or never. And everything that relied on the old rule quietly goes wrong: answers people were given, other docs, CRM fields, dashboards.
Search doesn't fix this. Search finds a document. Maya's question is a path: meeting → decision → who decided → the change that implemented it → the section version that changed → everything that depended on it.
That is a graph problem, so I built it as one.
What HandbookBrain does
HandbookBrain keeps one versioned knowledge graph in FalkorDB. It holds a company's documents, the meetings and chats where decisions are made, the merge requests that implement them, and the systems, CRM fields, metrics and dashboards those documents touch. An agent works on top of it:
- Answers "how does X work, and why?" with a citation to the exact version of each section it used, plus the trail back to the meeting, the person who decided and the merge request that wrote it down.
- Detects what a change breaks. When a section changes, it finds every stored answer that cited the old version, judges from the diff whether the answer is now wrong, notifies the page owners and re-answers.
- Closes the gap between decision and documentation. A decision that no document reflects after three weeks gets a drafted update, routed to the page owner to approve.
The demo uses GitLab's public go-to-market handbook (Marketing Ops, Deal Desk, Sales Systems, Channel Ops, Marketing Analytics) and replays 74 real merged MRs from April to October 2026 in date order. The meetings and Slack threads around those MRs are synthetic: I generated them with an LLM, anchored to the real changes, and every synthetic file is labelled as such.
The data model
Three ideas carry the whole design.
1. Version sections, not documents. A page is split into sections. Every change creates a new Section node that SUPERSEDES the previous one, with valid_from / valid_to. Nothing is overwritten, so you can read the graph as of any date, and a rule that was deleted is still findable.
2. Answers are nodes. An Answer CITES the exact section versions it used. That one edge is what lets a later change find the answers it breaks.
3. Decisions are first-class. Each decision is extracted from a meeting, chat or ticket with the speaker and a verbatim quote, linked to the section it is about, and reconciled to the commit that implemented it by reading the diff.
Conversation <-DECIDED_IN- Decision -DECIDED_BY-> Person
Decision -IMPLEMENTED_BY-> Commit <-MERGED_AS- MergeRequest
Commit -CHANGED-> Section -SUPERSEDES-> older Section
Page -HAS_SECTION-> Section -MENTIONS-> System | Field | Metric | Dashboard
Answer -CITES-> Section Answer -ANSWERS-> Question
Answer -INVALIDATED_BY / REVALIDATED_BY-> Commit
The Cypher that does the work
"Why is it like this?" is one path from a section back to the meeting:
MATCH (s:Section {sid:$sid})<-[:CHANGED]-(c:Commit)<-[:IMPLEMENTED_BY]-(d:Decision)-[:DECIDED_IN]->(cv:Conversation)
OPTIONAL MATCH (d)-[:DECIDED_BY]->(by:Person)
RETURN d.statement, d.decided_at, cv.title AS meeting, by.name AS decided_by, c.ts AS implemented_at
"Which answers did this change break?" starts from the commit, steps back to the version each answer cited, and returns the old and new text for a diff-based judge:
MATCH (c:Commit {sha:$sha})-[ch:CHANGED]->(s:Section)
OPTIONAL MATCH (s)-[:SUPERSEDES]->(prev:Section)
WITH ch, s, CASE WHEN ch.kind = 'removed' THEN s ELSE prev END AS cited
WHERE cited IS NOT NULL
MATCH (a:Answer {status:'fresh'})-[:CITES]->(cited)
RETURN a.id, ch.kind, cited.text AS old_text, s.text AS new_text
"What else relies on what changed?" goes from a change to the fields and metrics it mentions, then to every other current document and its owners:
MATCH (:Commit {sha:$sha})-[:CHANGED]->(s:Section)-[:MENTIONS]->(e)
MATCH (e)<-[:MENTIONS]-(o:Section)<-[:HAS_SECTION]-(p:Page)
WHERE o.valid_to IS NULL AND o.path <> s.path
OPTIONAL MATCH (p)-[:OWNED_BY]->(owner)
RETURN labels(e)[0] AS type, e.name, p.title, collect(DISTINCT owner.handle) AS owners
Retrieval is hybrid, all inside FalkorDB: a vector index on section embeddings plus a full-text index on heading and text, fused with reciprocal-rank fusion and filtered to current versions. Having vectors, full-text and graph traversal in one database meant there was no second store to keep in sync.
The agent itself is a tool-calling loop over 10 parameterised, read-only Cypher tools. Every claim must carry a citation that a tool actually returned, which is checked in code. A reviewer running on a different model sees only the question, the evidence and the draft, and can send the draft back.
Maya's question, answered
Here is how the trial-threshold story plays out in the replay:
- May 21: real MR !19600 adds Trial Threshold Scoring to the Marketo page. HandbookBrain links it to a (synthetic) Marketing Ops meeting 13 days earlier.
- Jun 1: a new analyst asks "At how many points does a trial user MQL?" The answer is 25 points, citing that section version.
- Jul 17: the (synthetic) Marketing Ops weekly decides to raise it.
-
Jul 23: real MR !20534 changes
25→31. The Jun 1 answer is judged stale from the diff, the owners from CODEOWNERS are notified, and the answer is refreshed to 31, with the meeting, the person and the MR attached.
Maya asks one question and gets the whole path.
Checking the graph against independent sources
An answer engine you can't verify is a liability, so I wrote an audit that checks what the site shows against sources the agent never reads:
| Check | Source of truth | Result |
|---|---|---|
| Change dates, MR numbers, version chains | git log |
2562 / 2562 |
| Who decided, where, the quote | meeting transcripts | 21 / 21 |
| Document owners | CODEOWNERS | 122 / 122 |
| Decision → implementing MR | hidden labels | 0 wrong links |
The audit earned its keep. My first decision-to-MR judge was too lenient and linked three decisions to later, unrelated changes. Now only what actually changed in the diff counts.
Evaluating the agent, honestly
This is the part I learned the most from.
My first headline was "89% vs 0%" on why questions against a plain RAG baseline. It looked great, and it wasn't fair. When I checked fact by fact, plain retrieval found the meeting date and the person who decided as well as my agent did. It scored 0% only because the merge request wasn't in its data at all.
So I rebuilt the eval, following Anthropic's Demystifying evals for AI agents:
- code graders where possible, and a model judge only for open answers;
- questions that should be answered and questions that should be declined;
- 3 trials per question, with the LLM cache off.
I also added a stronger baseline, RAG + git log, which gets the whole merge log as text.
29 questions × 3 trials:
| HandbookBrain | RAG + git log | Plain RAG | |
|---|---|---|---|
| All questions | 87% | 66% | 59% |
| Names the MR that implemented a decision | 18 / 18 | 4 / 18 | 0 / 18 |
| Old / new value of a changed rule | 15 / 15 | 9 / 15 | 6 / 15 |
| Meeting date of the decision | 18 / 18 | 18 / 18 | 18 / 18 |
| Plain lookups | 70% | 70% | 70% |
What this says:
- Finding the meeting is a retrieval problem, and retrieval solves it.
- What the graph adds is the link from a decision to the change that implemented it and the version history, including rules that were deleted.
- Even with the whole merge log in its context, the model rarely matched a meeting to the right MR on its own.
- On plain lookups the graph adds nothing, and the agent is slower and more expensive.
The caveats belong in the post too. The questions are mine, from one dataset, and I fixed the agent after reading its failures on these same questions. Expect lower numbers on new questions.
The eval found real bugs:
- A section titled Trial Threshold Scoring was buried under a generic "MQL and Lead Scoring" section, so the agent answered 100 instead of 31. Fix: heading-match candidates in search.
- Wide markdown tables were misread by column. Fix: rows are linearised.
- Rules deleted from a page were invisible. Fix: search also returns older versions that said more, with the MR that removed them.
- A rule agreed only in Slack was presented as policy, with a handbook citation. Fix: undocumented decisions are now marked "only said in chat", and the agent has to say so.
What I learned
- Make the answer a node. Once answers cite exact versions, "what did this change break?" is a traversal, not a guess.
- Version at the section level. Whole-document versions are too coarse to tell which answer a change affects.
- Measure per fact, against a fair baseline. An aggregate score hid where the graph helps and where it doesn't. The honest table is more convincing than the flattering one.
- Audit LLM-built edges against something independent. The model that extracts a link should not be the one that grades it.
What's next
- Direct connectors (Slack, Zoom, Drive) instead of exports, and speech-to-text for recordings.
- CRM and BI metadata (Salesforce fields, Power BI / Tableau dashboards) as first-class nodes.
- Opening the merge request for an approved update, not just recording the sign-off.
If your team's rules live in five tools and nobody remembers why any of them exist, I'd love to hear how you handle it.
Built with FalkorDB, Python, FastAPI and LiteLLM. Most of the code was written with an AI coding assistant (Claude) under my direction and review.
Top comments (0)