Teaching a Company Brain to Say "I Don't Know"
How we built ZEHN Core, a knowledge graph over company documents that refuses to answer unless it can cite the sentence it got the answer from.
The problem: confident answers with nothing behind them
Every engineer has been here: you're about to change a service, and you need to know what will break. The answer exists somewhere — a Confluence page from last year, a PR description, a throwaway line in a meeting transcript — but nobody can tell you where.
The obvious fix is "put an LLM on top of the docs." We tried that first, and it fails in the worst possible way: quietly. Ask a question the documents don't answer and you get a fluent, plausible, completely unsupported paragraph. For change-impact questions, a confident wrong answer is worse than no answer at all — it's how outages get approved.
So we set ourselves one rule for ZEHN Core:
Every answer cites the sentence it came from. If it can't, it says "I don't know."
Here's what that looks like in practice:
> What is our AWS root account password?
I could not find anything in the indexed documents about the
AWS root account password.
[abstained] 10 evidence ids retrieved
Ten passages matched loosely. None of them answered the question. So it said so.
> Who owns the Redwood API?
According to "Go-Live Protocols: Cross-Org Communications &
Escalation", the Oncall SRE owns the live monitoring dashboard
that tracks Redwood API performance metrics (ev_79b8a8236dff1aa5).
[answered] citing 1 of 5 retrieved evidence ids
That evidence ID resolves to a chunk, a document and a character range. You can check it.
Why a graph, and why FalkorDB
"What could break if we change X?" is a graph question. It's a walk: X is used by Y, Y feeds Z. Vector search alone finds passages that mention X; it can't follow a chain of dependencies across documents.
We used FalkorDB with the FalkorDB GraphRAG SDK to ingest a corpus of Confluence pages, GitHub PRs, meeting transcripts and email. The key design decision was to make provenance part of the graph's shape, not metadata bolted on afterwards:
(Document)-[:PART_OF]->(Chunk)<-[:MENTIONED_IN]-(Entity)
(Entity)-[:RELATES {rel_type, fact, spans}]->(Entity)
Every relationship carries its predicate, the extracted fact, and the character spans of the source text. Going from "origin-mapper affects indexer" back to the exact sentence that says so is one hop of Cypher.
On top of that:
- Controlled traversal. The model never writes Cypher. Retrieval runs through a fixed set of named, allowlisted queries, capped at three hops, and a traversal policy decides which predicates may be walked at all.
- Separate graphs for experiments. FalkorDB's multiple named graphs let us run extraction variants and pilots in isolated graphs while our baseline graphs stayed untouched.
Three layers that stop hallucination (only one is a prompt)
- The prompt forbids outside knowledge. Necessary — and nowhere near sufficient.
- Citation verification. Every evidence ID the model writes is checked against the IDs the tools actually returned. One fabricated citation and the whole answer is refused. One real citation does not launder an invented one.
- Forced abstention. No valid citation, no answer — however confident the prose sounds.
We didn't trust this on faith. The test suite feeds the agent loop a model that misbehaves on purpose — inventing citations, answering without evidence — and asserts the answer is withheld.
Grading the graph's own claims
A path existing in the graph doesn't mean the documents assert it. LLM extraction is noisy: the model sees "the release pipeline will emit v2 indexes" and happily writes an AFFECTS edge, even though that sentence describes a plan, not a dependency.
So for every path we return, each hop is checked against its own source sentence and labelled:
| Label | Meaning |
|---|---|
| Source-backed | Every predicate on the path is supported by its sentence |
| Inferred hypothesis | The path exists, but at least one hop isn't asserted by its sentence |
| Insufficient evidence | Negated, hypothetical, or the entity binding failed |
In the UI, inferred hops are drawn as dashed edges with a warning marker. When we ask about the Redwood API, the graph finds twelve paths — and eleven are labelled inferred hypothesis. A system that wanted to look impressive would show twelve confident dependencies. Ours grades its own work.
The same principle applies to ambiguity. "Redwood Inference" is a team, a service and a customer in our corpus. Rather than merging them to produce a tidier answer, ZEHN Core shows all three and stops.
What we learned
1. The model's "fact" is not a quote. Graph extraction stores a natural-language fact on each edge. We measured it: 0 of 186 facts were verbatim from the source. They read like quotes, which makes them dangerous. We stopped ever rendering fact as a quotation and built a quote-first path that stores the exact sentence instead.
2. Audit your edges, not just your answers. A manual audit found roughly one in three sampled relationships was bound to the wrong target entity. We didn't hide that — it ships as a visible limitation with every investigation.
3. Know your schema's physical shape. In our graph, every edge is physically RELATES; the semantic type lives in r.rel_type. Querying :DEPENDS_ON returns nothing — and nothing looks exactly like "no dependencies." That taught us our most important engineering rule:
A failure must never look like an empty result. A database error and "nothing depends on this" are different facts, and the UI renders them differently.
4. Don't turn on what you haven't proven. We evaluated making CALLS edges traversable. It produced interesting chains, but we couldn't justify it on the evidence, so it stays off in production — and the active policy is shown live in the UI. If someone enabled it, the badge would say Evaluation mode.
5. Say what you haven't built. Our corpus is a 30-document subset across four source types; there are no ticket sources and no staleness detection yet. The UI's history page says "history isn't persisted in this build" rather than showing fake rows. A control whose action doesn't exist is worse than no control at all.
The result
ZEHN Core ships as a FastAPI backend and a React frontend with seven screens: an impact radar with a tabbed evidence inspector, an agent that streams its real tool calls while it works, an interactive knowledge explorer, and an observatory that reports API health and graph health as separate facts. It's backed by 959 backend tests and 63 frontend tests, including ones that assert it can't cite a source that doesn't exist.
The biggest lesson wasn't about graphs or models. It was that trustworthiness is a product feature you can see: a dashed edge, a refusal, a limitation listed up front. Users can't verify fluency. They can verify a sentence.
Built for the FalkorDB Graph Hacks hackathon with FalkorDB and the FalkorDB GraphRAG SDK.
Code: https://github.com/Manav9547/company-brain
Top comments (0)