DEV Community: Mihir Rabari

I Gave a Fraud Agent No Vector Store and No Cypher. It Remembered Anyway.

Mihir Rabari — Sun, 18 Oct 2026 19:35:16 +0000

I Gave a Fraud Agent No Vector Store and No Cypher. It Remembered Anyway.

LedgerLens is three days old, and it already has a history. The seed data includes a closed case dated eight days back: a ring of accounts sharing device fingerprints. Today a shared-device alert lands in the queue, and before the agent touches a single transaction, it searches its own memory. It finds the seeded case, sees the overlap, and tells the investigator this isn't a new ring, it's a recurrence. It cites the case with the date it was learned and takes a different path through the investigation.

The history is planted so the tests can check the agent finds exactly what I put there. What matters is how it found it: a fulltext query over Finding and Case nodes, in the same FalkorDB graph that holds the bank's accounts. No vector store. No sidecar database. No retrieval layer bolted onto a chatbot.

This post covers the design, three rules that keep the agent honest, and the one bug that taught me that restraint also has to be remembered.

The stack: FalkorDB v6.0.2 (on Redis 8.10.2), FastAPI, glm-5.3 as the model. Three synthetic banks (bank_aurora, bank_nimbus, bank_verdant), each about 192 persons, 280-286 accounts, and 1,133 transactions, with two planted fraud rings and one planted negative control per bank.

One graph per bank, and nothing else

Fraud questions are rarely lookups. "Which accounts are linked to this flagged transaction?" is a walk through shared devices, addresses, owners, and money edges. Text retrieval guesses relevance from fragments. A graph follows connections that were actually recorded, and when an agent freezes an account, the path it walked is the explanation it owes compliance.

So the graph is the whole context layer: data, memory, SOPs, and audit trail. One FalkorDB instance hosts one graph per bank. Isolation is the graph key, since every handle is select_graph("bank_<slug>") with the slug validated against a strict pattern. A query can't cross tenants because there's no way to name another one.

Rule 1: the model never writes Cypher

If the agent writes its own queries, its answers depend on code nobody reviewed. So it gets a fixed menu of 26 tools, and a human wrote the query inside each one. CYPHER.md is generated from the live tool registry, so every query the product depends on is published for review.

Ids are checked before anything runs. validate_entities verifies arguments against the graph. A wrong id comes back as unknown_entity rather than a confident guess.
Reads can't write. Every read tool goes through GRAPH.RO_QUERY, enforced in the runner.
Writes leave a trail. freeze_account and escalate_case create an Action node with actor, reason, and timestamp, linked to the case.

Compliance policy lives in the graph too, chunked and fulltext-searchable (POL-14, POL-22, POL-31...). The agent is instructed to cite section ids in freeze reasons, fetching exact chunks like POL-31-C01, so every action points at text a reviewer can open.

Rule 2: every answer declares its evidence

Every session ends with a JSON verdict naming one of five answer shapes, and the tool layer rejects anything else.

Shape	What it tells the reviewer
`with_path`	The graph held the relationship; here are the hops
`from_memory`	A past session established this; here's when
`not_in_graph`	The entities exist, but nothing connects them the way the question assumed
`unknown_entity`	The id isn't in the graph, so no query ran
`action`	A deliberate, audited write

not_in_graph blocks "I couldn't find a link, so it's probably fine." In fraud work a missing relationship is itself a finding. unknown_entity blocks invented ids: no id in the graph means no query executes. Shapes don't make the reasoning correct. They make every claim say what kind of evidence it stands on, so a human knows what to check first.

Here's a real verdict from a fresh session asked about alert AUR-ALR-004. It took three turns (get_alert, search_memory, answer) and found 160 fulltext matches across findings, cases, and past alerts:

{
  "shape": "from_memory",
  "confidence": 0.9,
  "summary": "Investigation of AUR-ALR-004 concluded no_case (learned 2026-10-16T11:01:41+00:00): a single 12,400 transfer from Vera Petrovna (AUR-ACC-0940) to new beneficiary Zaid Reyes (AUR-ACC-0003) with no associations, no ring links, and no other suspicious flows; the alert remains open with no case attached."
}

(Names are synthetic. This verdict exists because of the bug in the next section.)

Rule 3: the account it must not touch

Every seeded bank contains one high-value, legitimate, completely isolated account, with a fresh device, a fresh address, and no ring links. It's the exact balance an over-eager agent freezes just to look productive. The investigator should examine it, find nothing, and return no_case. The live-LLM tests assert it stays untouched on every run.

The bug: the agent forgot every time it was right

While prepping the demo, I found a hole in the memory design. Every case the agent opened persisted as graph memory: findings, links, timestamps. But when the investigator correctly refused to act on the negative control (a legitimate 12,400 transfer), that decision left no trace in the graph. The graph only remembered cases we opened, never the ones we rightly refused. The next session would re-investigate the same innocent alert from scratch.

Discipline wasn't memory. An agent that has to re-prove an account's innocence every time is an agent that eventually gets it wrong once.

The fix (commit e93909f): clearances now persist as Finding nodes, linked to the session that produced them and the account they're about. It was 38 lines in agents.py plus a 27-line regression test. A brand-new session now recalls the refusal with the finding id and timestamp at confidence 0.9, which is the verdict above.

Algorithms are just more tools

FalkorDB's graph algorithms sit on the same menu:

algo.WCC for ring components
algo.pageRank and algo.betweenness for hubs and brokers inside a component
algo.SPpaths for up to three shortest money routes, with leg amounts
algo.maxFlow for funnel capacity between two accounts

MATCH (a:Account {id: $from}), (b:Account {id: $to})
CALL algo.maxFlow({sourceNodes: [a], targetNodes: [b],
                   relationshipTypes: ['TRANSFER'], capacityProperty: 'amount'})
YIELD maxFlow
RETURN maxFlow

These algorithms want direct node-to-node edges, but I model every transfer as a Transaction node because the audit trail matters. So the graph keeps two projections alongside the real model:

TRANSFER aggregates transactions per ordered account pair (amount summed, count kept). One edge per pair also satisfies algo.maxFlow's simple-adjacency requirement.
LINKED_TO records that two accounts share a device, an address, or an owner.

Rings come from LINKED_TO plus internal money density, because a household sharing a laptop is a connected component too. Size and internal flow separate a ring from a family.

A v6 gotcha: the algorithm API isn't uniform. Most procedures take a config map (algo.WCC({nodeLabels: ['Account'], relationshipTypes: ['LINKED_TO']})), but algo.pageRank still takes positional arguments (algo.pageRank('Account', 'TRANSFER')). I mapped the signatures against the running server instead of trusting the docs, and the working forms are in CYPHER.md.

Memory is just more graph

All three kinds of memory are nodes and edges.

Episodic: Session nodes with persisted Message chains (SAID_IN, NEXT). Each case gets its own session.
Semantic: Finding nodes with confidence and timestamps, linked to the entities they describe and fulltext-indexed next to case titles and alert summaries. search_memory is how both the opening scene and the clearance recall work.
Procedural: Playbook and PlaybookStep nodes hold the SOP retrieved before acting on each alert kind. Closing a case can call learn_lesson, which appends a step to the playbook, sourced to the case that taught it. Every later investigation follows it, and every change has a paper trail.

Recalling a prior session is just a query. There's no second store to keep in sync.

Handoff as an edge

Two agents run the same loop with different privileges. The intake analyst triages, reads anything, and opens cases, but can't freeze. The case investigator can freeze, escalate, and close, but only after a handoff. open_case triggers it by writing a :HANDS_OFF edge from the analyst's session to a fresh investigator session. Coordination becomes something you can query, not a function call that vanishes.

The numbers

I ran four investigations in parallel against the live stack:

Alert	Verdict	Tool calls	Wall time
AUR-ALR-001	no_case (refused)	11	120.3s
NMB-ALR-001	froze_and_escalated	37	221.4s
NMB-ALR-003	froze_and_escalated	54	265.1s
AUR-ALR-003	froze_and_escalated	62	408.6s

Wall time includes LLM inference. So it takes about 11 tool calls to refuse and 37-62 to take down a ring.

Query latency, median of 25 runs against the live FalkorDB:

Query	Median	p95
`search_memory` (fulltext)	0.37 ms	0.52 ms
3-hop money-path traversal	0.53 ms	0.70 ms
1-hop association links	0.58 ms	0.72 ms
`algo.pageRank` over TRANSFER	1.94 ms	2.85 ms
`algo.maxFlow` funnel capacity	46.9 ms	57.8 ms
`algo.WCC` (whole graph)	397.5 ms	498.2 ms

Memory recall is a sub-millisecond query, so it can run on every step of the loop at effectively no cost. The model is the bottleneck, not the graph. WCC is the heaviest because it labels the entire association projection, which is why it runs once per investigation and not per step.

Tests

44 pytest tests pass, including 3 live-LLM agent tests and 9 auth tests. Each bank has planted ground truth in tests/fixtures/planted_<tenant>.json, and the tests assert the tools find exactly what was planted and nothing more. For concurrency, the suite runs 24 parallel reads plus concurrent case and finding writes alongside the 4-way parallel investigations. I saw zero cross-talk and used no application-level locks. The MCP server exposes the same tool registry over stdio, so any MCP client can drive it.

Temporal edges, because recurrence needs time and not just pattern, and feeding algorithm output into alert prioritization.

Code: github.com/Mihir-Rabari/ledgerlens. Clone it, start FalkorDB, run python seed.py --wipe, and the console walks an alert from triage to verdict with the hops behind every conclusion on screen.

How Not To Rank a Hackathon

Mihir Rabari — Thu, 08 Oct 2026 16:11:06 +0000

Every hackathon I've entered, somebody else decided my fate. This one made me build the judge first.

DogFood Hackatho, run by Hackathon Raptors, gives every team the same 72-hour job: build a submission-and-judging platform against one published spec, one anonymized dataset, and one acceptance suite. Then the twist, straight from their brief: "the winning project is the one we run." Your code accepts your own submission, scores it, proves the score wasn't forged, and prints your certificate. The platform that judges you is the thing you shipped.

We were two people: Manas on backend, data model, and the judging math. Me on frontend, voting, the embeddable widget, the threat model, and the demo video. At the deadline we had all four spec tiers plus all four bonuses, a single docker compose up to a seeded portal, 548 passing tests, a 7/7 acceptance report, and a collection of scars worth more than any of it. The best ones:

1. The architecture decision was deleting most of the architecture

Day one we drew the stack that production instinct tells you to draw: seven Docker services. Postgres, Django, Next.js, nginx, then Prometheus, Alertmanager and Grafana on top. That's what a mature system looks like, right?

Then we reread the acceptance checker. It's vendored byte-for-byte from the spec's appendix, we diffed it to be sure (8,855 characters, identical), and it does exactly seven HTTP checks against a portal that must be up in 30 seconds. Grafana is not involved. Nobody asks for your dashboard. No panel has ever mattered less.

Mature systems have observability. Acceptance judges don't. We deleted the observability layer entirely, then went further and folded the remaining four services, Postgres, Django, Next.js and nginx, into one container with supervisord as PID 1.

If you self-host this in production, split it back out. That's not what it's for. It's for the organizer who clones the repo on a laptop five minutes before judging and needs a seeded portal, not an orchestration lesson. We traded some debuggability for an install story with zero moving parts. The customer is a stranger with a timer. Design for the stranger.

2. The scoring algorithm we almost shipped

The spec says judges score projects against a weighted rubric, then asks how you rank the results. The obvious implementation is the arithmetic mean. One line. We had it working.

Then Manas asked the question that killed it: what happens when one judge is systematically generous and another grades like a machine? Raw averages mix project quality with judge calibration, and the lenient judge drags every project they touched up the table. "We averaged the scores" is an answer. It rewards whoever was kindest.

So we fit a two-way additive model instead:

y_ij = μ + b_j + q_i + ε_ij

b_j is judge j's bias, q_i is project i's quality. Fitted by alternating means, Gauss-Seidel style: start with every bias at zero, compute qualities as the mean of de-biased scores, compute biases as the mean of de-qualified scores, recenter so the biases sum to zero, repeat until it settles. Every mean is taken only over cells that actually exist, because real score matrices are full of holes.

The edge cases were the real work. A judge who scores everything identically carries zero information: their bias reenters as zero and we flag their leverage instead of silently dropping them. A disconnected score graph (judges A saw projects 1-3, judges B saw 4-6, no overlap) gets is_connected: false and the API refuses to publish a ranking, 422, rather than invent one. Refusing felt like failure for about an hour. Then we realized you can't un-publish a winners list.

And we committed the proof: normalization-proof.txt in the repo, a rank-movement table on the fixture dataset showing bias shrinks and winners don't shuffle. For the pairwise bonus, a Bradley-Terry MM estimator with a phantom prior and a recovered-ranking test. Here's the industry punchline: Devfolio advertises "automatic score normalization" as a headline feature and publishes nothing about how it works. If the math decides who wins, the math is the product. Show it.

3. Three Docker gotchas

Postgres identifiers and the rebrand. We renamed the project mid-build, and a bulk rename turned POSTGRES_USER=dogfood into POSTGRES_USER=hack-hamster. Postgres refuses: SQL identifiers can't have hyphens. Identifiers stayed underscored (hack_hamster), the brand stayed hyphenated. SQL syntax outlives brand decisions.

The nginx Host header. With proxy_pass http://django_backend, nginx forwards the upstream name as the Host header by default. Django's ALLOWED_HOSTS check rejected Host: django_backend and the entire API 400'd behind the proxy while working perfectly when hit directly. One proxy_set_header Host $host; per location. Defaults are upstream-name-shaped, not request-shaped.

Webhook delivery threads. First version delivered webhooks from a hidden thread pool: a queue, two daemon workers, visible to no one. Worked great in dev. But a self-hoster inherits a worker process they didn't ask for and can't see, and "it works on my machine" is not a delivery guarantee. We replaced it with a synchronous single attempt plus a manage.py flush_webhooks batch retry. Explicit beats invisible.

4. The spec sentence that wasn't simple

"CSV export at every stage." We read it as one endpoint with an event_slug query parameter, built it, moved on. The spec means one CSV per stage: we ended up at six endpoints (scores, roster, assignments, submissions, audit, rankings). Annoying at the time, correct in hindsight. Six is more cacheable, more testable, and more honest about what an organizer is actually downloading.

The sneakier trap was the demo config. Our first seeding flow printed random session tokens and told you to paste them into the platform config file. Every fresh Docker volume invalidated it, which means the config we shipped worked on exactly one machine on Earth: ours. The fix was deterministic demo tokens, HMAC-SHA256(DJANGO_SECRET_KEY, "{label}:{email}"), derived from things that don't change across volumes. Never a database primary key; fresh volumes renumber those. New rule we live by: never bake runtime-ephemeral values into committed config, and never derive committed values from row order.

5. Role isolation is a backend property

Three roles plus anonymous, and it's very easy to enforce the boundaries in the UI and feel done. We treated it as a backend property and walked the whole thing over raw HTTP: 48 cells of role-by-endpoint matrix. Organizer hitting judge endpoints: 403. A judge asking for another judge's scores: 403 (that's the graded cell in the spec, peer scores stay sealed). Participants can't touch the CSV export. Anonymous: 401 everywhere except the public gallery. Zero mismatches, committed as a generated matrix file so it can't rot. Sessions are HMAC-signed, so a tampered cookie fails verification instead of failing open. Enforcement lives in the permission layer, not in components. If it's enforced in the UI, it's a suggestion.

One bug from the final hours belongs to the same family: demo account password hashes were getting clobbered by the boot-time fixture re-import, so the committed demo logins died on any fresh volume. Anything that must survive a fresh volume can't depend on insertion order.

6. How the endgame actually ran

The rules say AI tools are expected, so here's the honest version. The last night we ran about ten coding agents in parallel against the issue tracker: small, single-purpose issues, one branch each, a human merge queue into dev, no agent ever merged its own PR. They were good at exactly what you'd hope (pinning tests, docs conformance, port mismatches, small endpoint bugs) and useless at spec judgment, which stayed with us.

The part that made it survivable had nothing to do with AI. From hour zero we ran a conflict-zero contract: no file has two owners. I never touched config/, Manas never touched web/. Disjoint edits, zero merge conflicts, which means no tired person at hour 60 resolving a conflict badly and quietly breaking something that was green an hour earlier. We also froze the seam, the .hack-hamster.toml file the acceptance checker reads, at hour 20. After that, neither half of the team could break the other.

At hour 68 I was recording the demo video while the last fixes landed behind me. One full event lifecycle, submission to signed certificate, 2 minutes 27 seconds.

7. The receipts

My favorite line in the repo is in acceptance-report.txt. The spec's seven checks only machine-verify tiers 1 and 2. We built all four tiers. So the report we committed for judging reads, in full honesty: claimed but not verified: T3 T4.

We could have written "all tiers verified." Almost nobody would have checked. The organizers' own rule 06 is "claim your tiers honestly."

Here's what that line bought us: once a report admits what it can't prove, everything else in the repo reads differently. The 548 tests. The role matrix. The normalization proof. The conformance check that refuses to ship if a single route is undocumented. A repo that admits one gap is a repo you can trust on everything else.

If you want to check our work

Everything is in the open repo: choksi2212/dogfood-hackathon. Ten-minute reading order: .hack-hamster.toml (what we claim), acceptance-report.txt (what the checker verified), ARCHITECTURE.md (why the shape is what it is), JUDGING.md (the math, defended), THREAT-MODEL.md (the attacks we assumed).

The platform that judged us forced a kind of honesty a normal side project never demands: you can't bluff code that has to run against itself. If somebody forks it and runs a real event on it, that's the actual win.

Built at DogFood 2026 by Hackathon Raptors. Team Hack Hamster: Manas Choksi (backend, data model, judging math) and me, Mihir Rabari (frontend, voting, widget, threat model, demo video).