When checkout fails at 2 a.m., the error almost never names the cause. The failing endpoint is in one place; the setting that broke it sits somewhere else, reached through services nobody remembers wiring together. FAULTLINE is my answer to that gap: a prototype that shows what failed, checks evidence for a missing connection, and saves that connection in a self-run HydraDB graph so the next investigation starts smarter.
I built it for the November HydraDB hackathon, Track 1: a graph that improves through evidence-backed investigations. This post covers the problem, the hard parts, how the system works, what the measurements show, and where the claims stop.
- Live demo: faultline-j7kn.onrender.com
- Code: github.com/ParthMalhotra07/Faultline
- Walkthrough video: Loom
The problem: the visible error is not the cause
Incident response is mostly a search problem. An engineer sees a failing check, such as checkout returning errors, and has to work backwards: which service handles it, what that service depends on, and which recent change touched any of those dependencies.
The painful cases involve shared settings. A currency configuration changes. Checkout fails. A service map shows checkout and pricing, but it does not show that pricing reads that setting, because nobody ever recorded the link. So the team rediscovers the same hidden dependency during every incident, and the knowledge disappears again once the incident closes.
Two things make this hard for a tool to help with:
- Missing edges. The map the tool searches is incomplete by definition. A tool that only follows known links cannot explain a failure caused by an unknown one.
- Trust. If a tool guesses, engineers stop believing it after the first wrong answer. A useful tool must say when it does not know, and show the evidence when it does.
This is also the question the hackathon asked: can a graph earn its place over a plain vector store? A graph earns it only if it keeps structure (who calls whom, who reads what) and if it gets better each time it is used.
Challenges I faced
The interesting engineering was less about drawing a graph and more about making every claim the system makes defensible.
1. Making "I don't know" a first-class answer. A ranking model always returns something. FAULTLINE had to return insufficient evidence when no committed path connects a failing endpoint to a changed setting, and it had to keep abstentions, ambiguity and errors in the scoring denominator so they could not hide behind a flattering accuracy number.
2. Stopping the learner from cheating. In a controlled demo it is easy to leak the answer. I split the system: a private fault controller creates incidents and knows the true origin, but it hands that label only to the evaluator. The learner sees public observations and catalog metadata. Learning data and held-out data follow separate paths, and the protocol is frozen before the final run.
3. Keeping graph writes trustworthy. A graph that learns can also learn wrong things. A claim cannot be committed without a matching evidence ID, source hash and source version. Retries use deterministic IDs so they stay idempotent, and a failed or incomplete inspection cannot quietly improve the graph. Each learning round is also bounded: one source, 10,000 bytes, 100 records and a 1,000 ms budget.
4. Failing loudly when the graph engine misbehaves. The adapter caps requests at 5 seconds and 1,000 returned rows. A timeout, a truncated page or a malformed response becomes an explicit error, never an empty result that looks like "no path found". I run a pinned, open-source HydraDB image and keep its query port on container loopback, so the browser only ever talks to the FAULTLINE server and no arbitrary query endpoint exists.
5. Ambiguity between near-identical causes. Early on, a shorter graph path could win by default. When two changed settings cover the same failed and healthy endpoints, the data alone cannot separate them. I added a guard so the live demo now reports ambiguous instead of picking one. I added it after the frozen evaluation, so I make no separate held-out accuracy claim for it.
6. Comparing fairly against a vector store. I ran a retrospective vector experiment (local Qdrant with BGE-small embeddings) on the same acquired records. It is not information-equivalent, because the graph also held eight seeded topology claims that the vector corpus lacked, so I label it exploratory and do not claim it beats vector search in general.
7. Hosting a stateful graph on a free tier. The public demo runs on Render's free plan, whose disk is ephemeral. Graph state can reset after inactivity or a redeploy, and the first request after a quiet period can be slow. Persistence was verified locally with a mounted Docker volume, not on the hosted instance.
The solution: a graph that earns its keep
FAULTLINE runs a small commerce app (checkout, refunds, renewals) with real acceptance checks. It breaks that app on demand, searches a HydraDB graph for the cause, and, when the graph has a gap, inspects one bounded evidence source and writes the missing link back.
How the graph is shaped
HydraDB stores three node types: Entity, Claim and Evidence. Every dependency is a reified claim, a node of its own, so each link can carry its proof:
-
Claim -[:FROM_ENTITY]-> Entity(for example,service:pricing) -
Claim -[:TO_ENTITY]-> Entity(for example,config:currency) -
Claim -[:SUPPORTED_BY]-> Evidence(a source hash and version)
Endpoint-to-handler and service-call claims come from the app's versioned topology. Config-read claims, such as "pricing reads the currency setting", enter only after a source-backed inspection.
The components
| Component | What it does |
|---|---|
| Executable sandbox | Runs checkout, refunds and renewals; produces checks and bounded config-access telemetry |
| HTTP boundary | Serves the UI and public observations; enforces request fields, Host and Origin rules |
| Graph boundary | Sends allowlisted, parameterized queries to HydraDB; turns timeouts and truncation into explicit errors |
| Diagnosis | Traverses committed paths, ranks recent changes, keeps abstention and ambiguity visible |
| Ingestion and learning | Picks one permitted source, checks the budget, validates records, commits claims with provenance |
| Evaluation | Injects faults privately and scores frozen, adaptive, random and ablated graph snapshots outside the learner |
The investigation loop
- Trigger fault. The sandbox breaks a flow, such as checkout after a shared currency change.
- Investigate. FAULTLINE traverses committed, evidence-linked claims in HydraDB. If no path connects the failure to a recent change, it answers insufficient evidence.
-
Inspect & learn. The engineer chooses to let FAULTLINE inspect one permitted telemetry source within budget. It validates the records and commits new
ClaimandEvidencenodes. -
Re-diagnose. The updated graph now returns a supported path, with claim and evidence IDs, to
config:currency. - Reuse. A later incident that needs the same relationship follows the saved path with no new inspection.
The engineer stays in charge throughout. A person triggers the fault, decides when to inspect evidence, and chooses what to fix. FAULTLINE produces hypotheses, not repairs.
How it solves the problem, step by step
The first incident: checkout and the currency setting. The shared currency setting changes and checkout fails. The initial graph knows checkout and pricing, but not that pricing reads the setting, so FAULTLINE abstains with insufficient evidence instead of guessing. After Inspect & learn, one bounded telemetry source confirms the configuration reads. HydraDB returns the new claim and evidence paths, and diagnosis selects config:currency with the supporting claim and evidence IDs attached.
The second incident: refunds. The demo then switches to a refunds transfer incident on a different monitored endpoint. It needs the same pricing-to-currency link, and FAULTLINE finds it in the graph without inspecting anything. The first investigation paid for the second.
What the graph adds over text retrieval. The question FAULTLINE answers is structural: can a recent change reach each failing endpoint through specific service dependencies, while avoiding healthy endpoints, and which evidence supports every link? HydraDB performs that bounded traversal. A text index can find relevant records, but it would still have to rebuild and verify the endpoint, service and config connections to give the same explanation.
What I measured
The protocol is frozen. Learning processes six incidents; the final evaluation uses twelve distinct controlled fixtures and three fixed seeds, so each point pools 36 predictions. The primary metric is correct top-ranked origin divided by all attempted cases, so abstentions, ambiguity and errors count against the score.
| Method | Before learning | After 1 incident | After 2 incidents | Final round |
|---|---|---|---|---|
| Gap-driven HydraDB acquisition | 0/36 | 24/36 | 36/36 | 36/36 |
| Equal-budget random acquisition | 0/36 | 16/36 | 36/36 | 36/36 |
| Frozen graph (no learning) | 0/36 | 0/36 | 0/36 | 0/36 |
| Same-evidence token overlap | 0/36 | 0/36 | 12/36 | 12/36 |
What to take from it:
- Learning is the lever. A frozen graph never recovers; the learning graph goes from 0/36 to 36/36.
- Choosing the source matters at first. Gap-driven acquisition reaches 24/36 after one incident against 16/36 for random. Once both available sources are known, the two tie.
- The relationship matters. Removing the pricing-to-currency claims in an isolated final graph drops 36/36 to 24/36.
- Structure beats a simple text baseline here. The token-overlap baseline stalls at 12/36, and the exploratory vector run also reached 12/36 after acquisition, with the information-equivalence caveat above.
The selected run holds 1,044 prediction records, and an exact-protocol rerun matched its structural predictions and aggregate counts. The raw predictions and verifier scripts are in the repository so anyone can inspect the counts.
What I learned, the limits, and what comes next
Lessons. Provenance is the product: a connection with a source hash and version attached is something an engineer can check, and one without it is a rumor. Abstaining is a feature, because a tool that admits a gap earns the right to be believed when it does answer. And separating the learner from the scorer takes more discipline than any model choice.
Limits, stated plainly.
- Faults run on one fixed topology with only two telemetry sources, so knowledge saturates quickly. These are controlled cases, not a general accuracy claim.
- A reachable path supports a hypothesis worth investigating. It does not prove causation, and FAULTLINE does not repair anything.
- It is a single-writer prototype with a shared public demo sign-in, not an authenticated multi-user service.
- No customer interviews, willingness-to-pay results or production time savings are claimed.
Next. The natural first buyer is a small platform team that keeps rediscovering shared dependencies during incidents. A shadow-mode version could read approved, exported telemetry and service metadata and show evidence-backed paths for an engineer to review. Getting there needs durable storage, controlled data connectors, access control and validation on real incidents.
Try it. Open the live demo, choose Open workspace, then Reset experiment graph, Trigger fault, Investigate, Inspect & learn. The free host can take a moment to wake after inactivity. The full code, evaluation protocol and raw results are on GitHub, and the Loom walkthrough shows the whole loop in a few minutes.
Thanks to the HydraDB team for the open-source engine and the hackathon prompt that shaped this project.
Top comments (0)