TL;DR: We built an agent that investigates card fraud on a TigerGraph knowledge graph. It gathers evidence with GSQL queries, asks the cardholder when the evidence is thin, never makes up the answer, and only recommends actions the bank's policy allows. On 50 closed cases we never tuned against, it missed none of the 40 fraud cases, named 37 of 40 fraud patterns correctly, and called all 10 cleared cases legitimate.
Fraud analysts rarely miss fraud because they can't see it. They miss it because the job is slow. They pull the transaction history, trace the connected accounts, check devices against old cases, read the policy, and only then decide. By the time the case is written up, the money is often gone.
We built an agent to do that legwork, on TigerGraph, for the HHGOA challenge. We expected the hard part to be detection. It wasn't. The hard part was getting the agent to be honest about what it didn't know:
- Is a crowd of cards on one device really a ring?
- What do you call a pattern that fits none of the five documented ones?
- And the one that cost us most: what would the cardholder have said, when the dataset never tells us?
What we built
You give the agent a trigger (a risk score, a customer report, or an analyst request), and it works the case through to an action it can defend.
It opens a case and resolves the trigger to a card. It reads the transaction history, looks for rings of cards that share a device, region or email, and pulls up prior cases. Then it weighs competing hypotheses, each with a probability, and decides whether it knows enough to act.
If the policy says to ask the cardholder first, it asks. The dataset has no replies, so the agent writes down that none came back and re-recommends under the policy's "no reply" rule. It files a suspicious activity report when the policy calls for one, writes the case back to the graph, and explains every step.
Every tool takes an as_of timestamp and ignores anything after it. A case opened on 12 November can't see 13 November. We enforce that inside the queries instead of trusting the caller, because in a fraud benchmark a time leak looks exactly like good performance.
The stack:
- TypeScript end to end: pnpm workspaces, Turborepo, zod at every boundary, vitest, and a Next.js analyst UI.
- GSQL for the graph logic, on TigerGraph Community Edition 4.3 in Docker.
- The official
tigergraph-mcpserver as the only way the agent reaches the graph. No direct driver. - A local model for the assessment: Qwen2.5-7B-Instruct (Q4_K_M) on llama.cpp, at temperature 0, with its reply constrained to a JSON schema.
The architecture
The agent is an explicit state machine, not a free-running loop:
TRIGGERED → CASE_OPENED
→ INVESTIGATING GSQL queries through the TigerGraph MCP server
→ ASSESSING competing hypotheses with probabilities
→ EVIDENCE_PLANNING ask the cardholder when the policy says so
→ AWAITING_EVIDENCE no reply exists, and none is invented
→ EVIDENCE_RECEIVED records "no reply"; rule R4 applies
→ DECIDING final actions, each citing its rule
→ APPROVAL_ROUTING auto, L1 (team lead) or L2 (fraud manager)
→ EXPLAINING narrative and suspicious activity report
→ MEMORY_UPDATE case written back into TigerGraph
→ DONE
Tool calls have a budget, evidence rounds are capped, and the stop rule is code. The model gets to propose. The code decides what's allowed.
Here's one place that shows up. The model proposes hypotheses and probabilities, but code caps its confidence by how many independent kinds of evidence it actually looked at: fewer than two caps it at 0.45, and two cap it at 0.65. A 7B model can't argue its way past that, and the prompt tells it so.
We added a second guard later. If a case rests on a single independent fraud signal, even a strong one like a detector firing at 1.0, the code holds the fraud probability just under the blocking line. That way rule R1 ("verify before you block on a weak signal") applies whatever number the model returned.
The code is split into workstreams that talk to each other only through frozen contracts:
| Workstream | What it owns |
|---|---|
contracts/ |
The tool catalog, the answer-file schema, and fakes. Frozen after the first milestone, so everything else codes against it instead of someone else's half-finished code. |
graph/ |
The schema, the loaders, and the official TigerGraph MCP server in a container. |
gsql/ |
The installed queries and graph algorithms (next section). |
rag/ |
GraphRAG: chunks and embeds the policy and fraud typologies, and hands the model summarized evidence instead of raw rows. |
agent/ |
The state machine above. |
policy/ |
The rule engine, permissions and approval routing. |
eval/ |
The benchmark runner, the backtest harness, answer validation and leak checks. |
How TigerGraph is used
For us the graph is the investigation, not a lookup table.
The schema covers cards, customers, transactions, devices, addresses, email domains, identities and fraud cases. A NEXT edge chains each card's transactions in time, so velocity and bursts are a short traversal instead of a scan.
The agent's graph tools are installed GSQL queries, reached through the MCP server:
| Query | What it answers |
|---|---|
resolve_trigger |
Which transaction, card, customer and identity a trigger is about |
get_entity_profile |
A profile of one card, customer, transaction, identity or device |
txn_history |
A card's or customer's transactions up to as_of
|
neighborhood |
The devices, addresses and recipient emails around a card, 1 to 3 hops out |
card_velocity |
How many charges, and how much money, in a recent window |
shared_rings |
Every device, address or email this card shares with other cards |
baseline_deviation |
How a charge compares with the card's own earlier history |
detect_patterns |
Scores for the documented fraud patterns |
community_lookup |
Which fraud community a card belongs to, if any |
find_prior_cases |
Earlier cases on this card whose outcome was known at as_of
|
get_pattern_profile |
A pattern's required evidence and permitted actions |
vector_search |
Cosine top-k similarity, inside TigerGraph |
upsert_case_record |
Writes the finished case back into the graph |
On top of those we run graph algorithms: weakly connected components, label propagation, shortest path and hub detection.
Detectors and rings
The five documented fraud patterns are GSQL detectors over the graph. We added a sixth for a pattern the dataset doesn't name (more on that below). Shared-entity rings, meaning the cards that touch the same device, billing region or recipient email in a recent window, are a traversal. We couldn't express them cleanly as a table query.
Community detection found a data artifact first
A naive weakly-connected-components pass lumped 274 cards into one "cluster". It turned out to be the busiest cards in the dataset bumping into each other through sheer volume, and its confirmed-fraud rate was below the dataset's baseline.
So we added an overlap guard: two cards connect only if what they share is a meaningful slice of each card's own activity, not just any shared value. That broke the blob down to a largest component of 41 cards and left three real communities, all at 100% confirmed fraud. Those communities and the shared-device rings are written back as Pattern vertices, so a cluster found once becomes evidence for the next case too.
Vector search and case memory
Vector search lives in TigerGraph as well: cosine top-k over 29 policy chunks and 5,565 closed-case embeddings, with no separate vector database. Finding similar prior cases and finding the relevant policy text use the same engine over the same store.
Case memory closes the loop. Each investigation is written back as a FraudCase vertex by upsert_case_record, and the next investigation's find_prior_cases call can find it.
The agentic capabilities
Competing hypotheses, not a single label
The assessor proposes several fraud types with probabilities that add up to one, and each cites the evidence for and against it. The verdict comes from probability bands whose edges line up with the policy's 0.70 blocking threshold.
Two scorers with separate jobs
Jev, a hosted decision model from TypeSafe, only re-splits the probability the assessor has already put on fraud across the documented patterns. It never decides fraud versus legitimate, and it never sees a case the assessor called legitimate.
We also wired in Kev, a Jev-like model family we can fine-tune and run ourselves, behind the same interface. Its training export stopped partway through, so we don't trust it yet. It's our path to a fully local pipeline.
Knowing when to stop, and when to ask first
The stop rule weighs how many independent kinds of evidence agree, what contradicts them, whether the intended action is even permitted, and how much budget is left.
If a case sits between roughly 0.40 and 0.85, the agent asks the cardholder before stopping, even when the stop rule would let it finish. The policy's own order is recommend, verify if needed, then recommend again, and we follow it rather than call the case early. uncertain is a perfectly good answer.
Asking for evidence honestly
The dataset has no cardholder or analyst replies. We used to simulate one and stopped (more on that below). Now the agent states its assumption plainly: the request went out and no reply came within 24 hours.
Rule R4 ("no reply within 24 hours") then drives the final recommendation, not a guess about what the customer would have said. The answer keeps both the first and the final recommendation, and says what changed and why.
Policy as code, with guards the model can't talk its way around
Every action carries an approval route. Only auto actions run; L1 and L2 actions wait for a person. Beyond the single-signal cap above, a few more rules live in code:
- A case opens once fraud probability reaches 0.30, exactly as the policy's "a case is not a report" section says.
- The "coordinated or repeated abuse" rule (R9) fires only when the abuse actually spans customers, not for a burst on one card.
- An online-flagged charge can never be labelled account takeover or out-of-region use, because the dataset defines both of those patterns as card-present.
Results
Nobody has the answer key for the 20 benchmark cases. What we do have is closed_cases_history.csv: 5,565 finished investigations with real outcomes.
We built a leak-free backtest from it. It replays each closed case as if it had just opened, with every tool seeing only what was true at that case's opened_at, and compares the agent's answer with the analyst's.
We report two samples. "Fresh 50" is cases we never used to tune or measure anything. "Original 50" is the sample we iterated against.
| Metric | Fresh 50 | Original 50 |
|---|---|---|
| Fraud pattern correct | 37/40 (92.5%) | 35/42 (83%) |
| Fraud cases missed | 0/40 | 0/42 |
| Cleared cases called legitimate | 10/10 | 6/8 (2 uncertain) |
| Cleared cases blocked | 0/10 | 0/8 |
| Verdict agreement with analysts | 50/50 | 50/50 |
| Report decision matches analysts | 46/50 (92%) | 44/50 (88%) |
| Exposure within 25% of the analysts' figure | 32/40 (80%) | 26/42 (62%) |
The history never shows a model alert that turned out to be fraud: every fraud case started as a customer dispute. So we also replayed confirmed disputes as the alerts they could have been. On 31 cleared alerts and 50 fraud-as-alert cases, all held out from everything else:
| Held-out alerts | Before calibration | After |
|---|---|---|
| Cleared alerts called legitimate | 0 of 45 | 18 of 31 |
| Cleared alerts blocked | 6 of 45 | 5 of 31 |
| Fraud alerts blocked | 32 of 69 | 25 of 50 |
| Probability error (1:1 Brier) | 0.304 | 0.160 |
4 fraud alerts were called legitimate, but none was closed or allowed.
With about 40 fraud cases per sample, a swing of two cases either way between runs is noise. The local model isn't perfectly deterministic under small input changes, even at temperature 0.
What we learned
A coin flip beat our entire graph
The dataset doesn't include cardholder replies and tells you to simulate them. We did, carefully. A hash of the request fields picked "customer confirms" or "customer denies". It was deterministic, documented, and never read a fraud label, so it wasn't leakage.
It was still made up, and it was a disaster. Every confirmed-fraud case that happened to draw "confirms" closed as legitimate: six out of six in one backtest. One of those cases had 159 pieces of graph evidence, and a single hash bit threw all of it away. Twice, in fact, because a second piece of code applied the same fake reply again as a separate veto.
We took the simulation out completely. The agent now records that no reply came, which is the only true thing we can say about a reply we don't have. It still recommends verifying with the customer, because the policy asks for that recommendation. The policy never asks us to invent the answer and reason from it.
Calibration was mostly us fixing our own evidence
On an early 20-case sample (17 of them confirmed fraud), pattern accuracy was roughly 41–47% with the 7B model alone, and it wobbled between runs even at temperature 0. Giving the pattern scorer calibrated, weighted evidence instead of raw text took it to 58.8%.
On a 50-case sample the same chase went 76.2%, then 78.6%, 81.0% and 83.3%, and each step was a bug where our evidence was quietly lying to the model:
- "This charge has no history" was really a 500-row history cap running out before it reached the right date.
- A card's own earlier fraud was being read as another card's fraud, which made too many cases look like rings.
- A baseline check reported a z-score of 1,110 from just two earlier transactions. That's a small-sample artifact, not a signal.
- A shared-device ring counted a card's own fraud from a month earlier as if the device were active now.
None of those were model problems. Each time, we handed the model a wrong fact and then acted surprised when it reasoned from it.
A subagent found the pattern we were missing
The dataset says outright that not every fraud pattern in it is documented. We sent a read-only subagent to look for one, and barred it from designing anything against our own backtest cases.
It found a second cluster inside the closed "undocumented" cases: four online charges within an hour, each in a narrow $400–$500 band, that none of our five detectors caught. We built a sixth detector for it and checked it against the closed-case history before trusting it. It now correctly names both closed "undocumented" cases in our backtest sample. Before, we could only mislabel them as a documented pattern.
Known limitations
Cleared alerts can't fully close
An analyst closes a false alarm after actually talking to the cardholder. We don't fake that conversation, so our agent gets as far as "asked, no reply, the policy says decline the flagged charge and monitor", but not to the analyst's clean CLOSE_NO_FRAUD. That's the direct cost of removing the fake reply, and we'd rather pay it than bring the fake back.
We did test a middle ground: assume a confirmation only when the evidence shows no independent fraud signal, and keep that rule only if it closed zero fraud. On 69 confirmed fraud cases replayed as model alerts plus 45 real cleared alerts, the best such rule closed 24 of 39 cleared alerts, but also 6 of 37 fraud cases. It failed the zero-fraud bar, so it isn't in the agent. A calibrated evidence model fitted on 605 alerts failed too: even at its strictest cutoff it closed 1 of 388 held-out fraud cases.
About half of fraud alerts are held, not blocked
Every confirmed fraud case in the history began as a cardholder dispute, so the history never shows a model alert that turned out to be fraud. Replaying disputes as the alerts they could have been, the agent blocked 25 of 50 and held the rest for verification and monitoring. 4 were called legitimate, but none had its transaction allowed or its case closed.
A legitimate verdict is still a judgement call
The local LLM put every cleared alert at 0.50 or higher. So on model alerts the fraud probability now comes from an evidence model fitted on 605 replayed alerts and checked on 629 held-out ones (AUC 0.909). Five facts drive it:
- whether the flagged charge was online
- whether the device is new to the account
- whether there are prior cases
- whether a device-sharing fraud signal fired
- whether the card has a cleared case
Surprisingly, a new device leans legitimate here (84% of cleared alerts against 24% of fraud), which fits the dataset's own caveat that "people buy new phones".
Six of our 20 answers are now legitimate, all at 0.37. On held-out alerts with exactly that evidence (11 cleared, 9 fraud), about one in three was fraud once both outcomes are weighted equally. So the verdict follows the likelier reading, and the actions hedge: they ask the cardholder and keep the case open instead of closing it.
Account takeover and out-of-region use can look identical
With what our tools can see, the two are sometimes truly indistinguishable. We searched every combination of up to three features the agent sees, on held-out cases. Even for the best one, the rarer pattern was only 30–42% of the cases it matched, so any rule that names the rarer pattern gets more cases wrong than right. A learned scorer reading the same evidence hits the same wall.
Exposure runs short on long episodes
This happens especially on card-testing runs that stretch over weeks rather than hours. Our episode window is tuned for the common case and undercounts the tail.
Jev is a hosted API call
Pattern re-scoring sends every assessed case out to TypeSafe's service. The local alternative, Kev, uses the same interface but isn't trained well enough to turn on by default.
We open cases the analysts never opened
The policy opens a case once probability reaches 0.30 or evidence is requested, and the agent requests verification on every uncertain alert. The rule has no "unless it later clears" exception, so we followed it as written instead of adding one.
What we would improve with more time
- Finish training Kev. Swapping an external API call for a small model we control and can inspect would help both latency and auditability, and the interface is already there.
- Keep pushing on the cases that look irreducible. The 30–42% ceiling holds for the features we extract today. Authorization and hold status, or a real multi-hop path feature instead of single-hop rings, might move it. We haven't tried yet.
- Calibrate beyond model alerts. The evidence model sets the probability only on risk-score alerts. Disputes and analyst requests still take the language model's number.
-
A real answer for cleared alerts. If a future version of the dataset, or a live deployment, supplies the cardholder's actual reply, it goes into the same
evidence_requestsrecord the agent already writes. The rules that use it (R2, R3, R4) are already built.
Built on TigerGraph Community Edition with the official TigerGraph MCP server. Every number above comes from closed cases in the provided history, not from the 20 benchmark cases. We can't know our score on those, and neither can anyone reading this before judging.
Top comments (0)