Dragnet: an agentic fraud investigator on TigerGraph

Swasthika Devadiga — Fri, 25 Sep 2026 05:49:05 +0000

Built for the TigerGraph Hacker House Goa challenge. Take 20 fraud alerts from the last two months of a six-month card ledger, investigate each against a graph and 5,565 closed investigations, work out what kind of fraud it is if any, and recommend an action under a written bank policy. Half the exam cases are legitimate, so an agent that blocks everything fails.

The shape of the system

UI (static SPA, app/frontend)  ->  FastAPI (app/backend/app.py)
investigator.py   trigger -> graph evidence -> GraphRAG retrieval -> score
                  -> R1..R10 decision -> evidence request -> re-score
                  -> explain -> write the case back to the graph
graphfactory.py   TigerGraphAdapter when the workspace answers, else
                  LocalGraphAdapter (sqlite), and it logs which and why
features.py / classifier.py / refit_head_v2.py / laya.py   scoring
corpus.py / retrieval.py / llm.py                          GraphRAG, narration

The graph is a TigerGraph Savanna workspace in ap-south-1, TG-0 (4 vCPU, 32Gi), TigerGraph 4.2.5, graph FraudGraph. Loaded and checked against expected counts: Transaction 590,742, Card 14,317, Customer 13,553, DeviceProfile 9,706, ClosedCase 5,565, BillingRegion 332, EmailDomain 60. Thirteen GSQL queries drive the investigation; twelve were installed and enabled at the time of writing.

Two schema decisions worth defending. The sparse Vesta columns (C1-C14, D1-D15, M1-M9, id_01-id_38) are stored as STRING, verbatim: typed numeric, an empty token becomes 0.0 and the agent cites "D1 = 0 days" for a value nobody measured. And identity data rides on the FROM_DEVICE edge, not the Transaction vertex, because an identity record describes a device session, not a payment. Every edge declares a REVERSE_EDGE: walking from a device back to its transactions is the traversal this task turns on.

The agent loop

Trigger. A risk score, a customer report or an analyst request. Nothing dated after max(flagged ts, opened_at) is read, so the agent cannot see its own future.
Graph evidence. Card timeline, device and region profiles, velocity, card-testing burst, shared-device cards, recurring match, and the closed cases on the card, the customer and the device. Each becomes an evidence[] entry with source: "graph" and a ref naming the query behind it. Retrieved prior cases, rule text and typology text come back the same way with source: "document".
Score. A calibrated probability, a 7-way pattern, a precondition veto, two undocumented detectors, and Laya as an optional disagreement signal.
Decide, ask, re-decide. R1 to R10 produce initial actions, each carrying its route and its rule. When R1, R4 or R8 hold, the agent issues one evidence request with an explicit assumed response, re-scores, and emits final actions plus what_changed.
Permissions. L1 for DECLINE_TRANSACTION and BLOCK_CARD at or below $2,500, L2 above it and for BLOCK_ALL_CARDS and FILE_REPORT. The agent recommends and routes; it never executes an approval-gated action.
Memory. The case is written back as a FraudCase vertex on the same edge types the closed cases use, so history queries read agent-written cases with no second code path.

The handset that was not diagnostic

The dataset ships nine closed cases labelled undocumented: confirmed fraud the analysts could not match to any documented typology. They are not one pattern. They are two.

Four (CC-2649, CC-2971, CC-2985, CC-3035, opened 27 Aug to 4 Sep 2016) are a device ring: a Samsung SM-G935F on Chrome for Android behind an anonymous proxy, worked across cards belonging to different customers, connected_card_ids union to 24 cards, exposure $108 to $390 each.

The other five (CC-3748, CC-3841, CC-3907, CC-4086, CC-4124, opened 20 to 29 Sep) are something else: four online purchases inside forty minutes, each just under $500, exposure between $1,871.13 and $1,922.65, connected_card_ids empty. Nobody is sharing a device. Somebody knows where a $500 authorization ceiling sits and splits one theft into pieces that each clear alone.

The obvious move is to treat the handset as the signal. It is wrong, and the graph says so. The closed cases that touch any SM-G935F profile through their transactions come to 15, across four patterns: 9 card_not_present_new_device, 4 undocumented, 1 card_not_present_fraud, 1 cleared alert. The Galaxy S7 Edge was a popular phone, and this data holds 24 distinct SM-G935F profiles.

The discriminator is the full profile plus the connection: SM-G935F Build/NRD90M | Android 7.0 | chrome 62.0 for android | 1920x1080. That profile carries 114 transactions across 52 cards and 52 distinct customers between 15 Aug and 4 Dec 2016, every one of them marked id_15 = New and id_23 = IP_PROXY:ANONYMOUS. Exactly four closed cases sit on it, and all four are the ring.

So the detector never fires on a device alone. It needs a specific handset, not a generic platform string, plus at least two other cards on it already carrying confirmed fraud. The commonest profile in this data reaches 842 unrelated customers, which is why q_shared_device_cards returns device_reach_cards and device_reach_customers next to the hit list.

The ring runs past the closed-case window into the exam. HHG-014's flagged transaction 3478561 (card C13487-K1, 2016-11-22 16:11, $74.96) sits on that exact profile with New and IP_PROXY:ANONYMOUS, on a card named in none of the four closed cases. The agent reaches undocumented at probability 0.83 and retrieves those four at the top of its precedent list.

A phrase that poisoned every query

The GraphRAG corpus is 5,613 chunks and 431,751 postings over 22,165 terms in a local BM25 index: the 5,565 closed cases, the 10 policy rules as separate chunks, and the README's typology and answer-format sections.

An early retriever queried by exemplar, paraphrasing the situation and matching it against the notes. The paraphrase for a disputed charge was "purchases they did not make". That phrase appears verbatim in 4 of 5,565 analyst notes, and those four are the device ring. It was not a query, it was a fingerprint. Every disputed-charge case pulled the ring to the top of its precedent list, including HHG-003, a $49 charge the cardholder had paid before at the same amount.

Two fixes. Each pattern now carries a discriminating sentence copied verbatim from the notes that actually separate it. And structured re-ranking applies a 0.65 multiplier when a candidate's pattern contradicts what the graph features imply. BM25 takes 0.55 of the final weight, structured signals the rest, and every matched component is returned in why_matched. On templated analyst prose, lexical retrieval finds the template, not the meaning.

Calibration, and a leak that looked like skill

The training pool is 84% fraud; the README says half the exam is legitimate. Platt scaling calibrates to the training prior, so inference applies one explicit offset in log odds, logit_exam = logit_train + logit(0.5) - logit(p_train). That moves the intercept only, never the evidence weights, and both probabilities are returned. On a time-ordered held-out block of 1,113 closed cases the fit never saw: AUC 0.9849, 10-bin ECE 0.0290 uncalibrated, 0.0242 after Platt, 0.0175 after the shift.

Leakage is the more instructive half. Outcome and trigger are perfectly confounded in the history: all 4,665 confirmed-fraud cases open with a customer report, all 900 cleared cases with a model score. Fit on that alone, a model learns the trigger and reads the bank's own risk score as evidence against fraud. So no feature encodes the trigger, and the negative class was widened with controls matched to the positives' risk-score distribution.

Necessary, not sufficient. The first control sampler drew only from cards with no confirmed fraud case, and only from August to October, so prior_cases_on_card was near-zero on controls by construction, and since every exam case is dated 12 Nov or later, the count was partly a clock. That head gave HHG-018 a prior_cases_on_card of 20, a +19.13 logit contribution and p = 1.0000 on a $39.08 charge the card had already paid at the same amount and merchant. A reliability curve would not have caught it. Reading per-feature contributions on the exam inputs did.

The head now in use draws controls from all cards and all months, stratified by month and risk-score decile, log1p's the counts, splits by time (train 6,393, calibrate 2,131, test 2,131, spanning 23 Oct to 31 Dec) and fits under 50/50 class weights. On that block: AUC 0.8808, weighted ECE 0.1288 before calibration and 0.0284 after. Lower AUC than the leaky head, and real. One honest weakness stays: undocumented has support 0 in every test split, since only 9 exist in 5,565, so that detection is rule-based, not learned, and the case files say so.

Two failures that looked like success

Eleven distinct bugs came out of the loading path. Two returned success.

The REST loader rejects eol '\r\n' with REST-30000, so we converted the shipped CRLF files to LF. The loading jobs still declared EOL="\r\n". A load ran in 4.9 seconds, returned no error, and loaded zero rows. The installer now measures every file's line ending at run time and prints any disagreement with the job's USING clause.

Second, /restpp/ddl ignores the job's HEADER="true" and runLoadingJobWithFile has no header argument, so every header row loaded as data. The graph held a Transaction with id TransactionID, EmailDomain vertices named P_emaildomain and R_emaildomain, and every vertex type sat at exactly +1 against expected. Headers are now stripped before upload.

The rest, briefly: BOOL DEFAULT FALSE needs a quoted literal; proxy is a reserved keyword, and the parse error pointed at the wrong token; multi-pair edges need | between FROM/TO pairs, not a comma; and GSQL 4.2 rejects a function call inside a vertex-set literal, so Seed = {to_vertex(id,"Card")} becomes VERTEX v = to_vertex(id,"Card"); Seed = {v}; in 14 places across 12 files.

The graph decides, the LLM explains

Claude Opus 5 writes case summaries, SAR narratives and action reasons. It never decides. investigator.py passes the decision to the model as authoritative and asks for the model's own assessment separately, so the engine can detect disagreement rather than absorb it.

That is a design choice, not a limitation. R1 says do not block on a single signal below 0.70. R2 and R7 split on whether a disputed charge matches the card's own recurring pattern. These are computable predicates over graph facts, and an LLM asked to apply them will apply them most of the time, which is not a control. _apply_r1 strips BLOCK_CARD and BLOCK_ALL_CARDS from any recommendation resting on one weak signal, and records that it did. _assert_consistent refuses to write a file where sar.file and FILE_REPORT disagree, or where a legitimate verdict carries affected transactions.

Laya (convaiinnovations/laya, Apache 2.0, 421M) is a second opinion by a different route: a non-autoregressive decision model that answers typed questions in one forward pass and generates no text, so there is nothing to parse and nothing to hallucinate. It does not average into the probability; disagreement raises uncertainty, pulling the decision toward VERIFY_WITH_CUSTOMER under R1 and ESCALATE_TO_ANALYST under R8. It loads in 0.6s, is gated behind HHGOA_LAYA, and the shipped answer files do not use it.

The split that cost the most thought

The ensemble returns two probabilities, and for a while we reported the wrong one.

Laya, a 421M open decision model, runs beside the calibrated classifier as an
independent second opinion. Where they disagree the ensemble nudges the number.
That adjusted figure was what the answer files carried as fraud_probability,
which is the field the brief says is scored for calibration.

Measured across the 20: mean absolute delta 0.028, maximum 0.130, and the
movement was systematically inflating confident-legitimate cases. Laya's own
checkpoint warns its confidence is uncalibrated, so we were degrading a
calibrated number with an uncalibrated one, in the field being graded on
calibration.

HHG-010 showed what that costs. The pull moved 0.24 to exactly 0.30, crossing
the legitimate threshold, and that single crossing cascaded into a different
verdict, a populated affected_txn_ids, $1,000.03 of exposure that did not
exist, and a pattern label. One rounding of one number, four downstream lies.

The fix is two tracks rather than one. p, the calibrated probability, is what
we report and what sets the verdict. p_dec, the disagreement-adjusted one,
only gates whether the agent asks for more evidence. The assessment stays
honest; the caution stays. Evidence requests did not change: still 13.

MCP, and where we deliberately did not use it

The TigerGraph MCP server exposes 69 tools against the live workspace, and it is
the best thing we touched all week. Asking it for device profiles shared by more
than six customers returns, in one call, a profile carrying 3,648 transactions
and 1,013 cards across 1,011 separate customers. Nobody wrote a screen for that
question.

The 20 graded cases do not run through it, on purpose. tool_calls in an answer
file has to be a count, not an estimate, so every call goes through one adapter
contract that counts itself. Generic MCP tools do not map one to one onto that
contract, and a run has to reproduce. MCP is for reaching past the contract;
the contract is for being graded.

What the agent produced, and what is not finished

Across the 20 exam cases: 10 legitimate, 6 uncertain, 4 fraud. Patterns: 10
none, 5 card_not_present_new_device, 3 undocumented, 1
card_not_present_fraud, 1 account_takeover. Two SARs, the right
neighbourhood given that only 397 of 5,565 closed cases filed one.

978 graph and retrieval calls, averaging 48.9 a case and varying with the
case. 254,307 model tokens. 876 seconds of wall clock across the twenty.

13 of the 20 asked for evidence before deciding, and in 9 of them the
recommendation changed once the answer came back. That number is the one we
would most like read, because it is the difference between an agent and a
scorer, and it is visible in the interface rather than asserted: each case shows
its initial and final action lists side by side, with additions tinted and
dropped actions struck through.

The validator runs 3,402 assertions across the 20 files and all of them
pass. Every id is checked for membership in the dataset rather than against a
regex, exposure is recomputed from the transactions each file claims, the L1
versus L2 routing boundary is checked at $2,500, and a SAR flag that disagrees
with its own action list is an error. An earlier run had 20 failures, all
tokens is 0, because the API key hit its spend cap mid-run and 16 files
shipped with deterministic summaries instead of prose. That is fixed: every file
now carries real model output.

Both undocumented typologies are found. HHG-014 is a shared-device ring across
nineteen cards. HHG-006 is threshold structuring, four charges inside thirty
minutes at $478.95, $456.96, $488.04 and $482.12, totalling $1,906.07, and it
retrieves all five structuring precedents from the closed-case history.

What is still not finished. Embeddings remain unavailable: no Anthropic
embeddings endpoint, no Voyage key, no sentence-transformers in the venv, so
every retrieval result is labelled mode="bm25+structured" rather than passed
off as semantic search. undocumented has support 0 in every test split, since
only 9 exist in 5,565, so that detection stays rule-based and the case files say
so. And written_cases_for_card() exists on both graph adapters with zero
callers, so cases are written to the graph and never read back: the memory loop
is built and not closed.

The pattern across all of it: the first plausible signal was wrong every time.
An SM-G935F looked like a ring and spanned four patterns and a cleared alert. A
vivid phrase looked like a query and was a class label. A high AUC looked like
skill and was a calendar. An ensemble looked like caution and was decalibration.
Each was caught by running a count against the graph, not by reading the code
again.

DEV Community: Swasthika Devadiga