Don't Trust the Score: A Fraud Investigator That Argues Both Sides on TigerGraph

Pranav S Salian — Thu, 24 Sep 2026 17:06:44 +0000

🎥 Demo video: Watch the Fraud Investigator in action

The finding that changed the whole design

Most fraud-agent demos work the same way: a model produces a risk score, then an LLM writes a convincing story about why the score is right.

Before writing any code, We profiled all five provided files with DuckDB. The data showed that approach would fail in both directions: it blocks innocent people and misses real fraud.

What I measured	Result	Design consequence
Cleared (false-alarm) cases	All 900 scored 0.81 or higher	A high score is where innocence lives, so the agent must be able to clear a 0.9
Confirmed fraud	Mean score 0.47, and 31% scored under 0.3	The agent must be able to convict a 0.05
A hidden fraud ring	60 transactions, 28 customers, every score ≤ 0.44, in no closed case	A score-blind sweep is needed
Graph features vs bank score, among alerts ≥ 0.8	AUC 0.91 vs 0.62	The graph decides where the score is confused
"Customers"	104 of them hold 48% of all transactions	A customer is not a person, so baselines must be per card

Here is the same story as one picture. Fraud appears at every score. Innocence appears in only one zone:

bank score bin : 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
confirmed fraud: ███ ███ ███ ███ ███ ███ ███ ███ ███ ███
cleared (safe) : · · · · · · · · ███ ███
                                                            ▲
                              the "decoy zone": the score cannot separate them here

Chart shows where each outcome appears; it is not a count.

The score tells you where to look. The graph tells you what happened.
So I built an agent that treats the score as one weak witness, not the judge.

What We built

Fraud Investigator takes an alert from any of three triggers (a bank risk flag, a customer saying "I never made this", or an analyst asking "look for related activity") and investigates it like a small legal team:

Prosecution runs fraud signatures: test-then-spend, off-profile burst, threshold hugging, shared rare fingerprint.
Defence runs innocence signatures: trip continuity, device succession, recurring cadence, baseline-consistent spend.
A calibrated judge weighs both sides into a probability.
A policy engine converts it into actions, and only it can.
Plain-language explanations for the customer, and a Suspicious Activity Report narrative when policy requires one.

The architecture

Five layers. Everything that reads goes through TigerGraph's MCP tool surface. Everything that can change the world goes through the policy engine.

flowchart TB
  subgraph ENTRY["ENTRY"]
    direction LR
    E1["Risk-score alert"]
    E2["Customer report"]
    E3["Analyst request"]
    E4["Analyst console"]
  end

  subgraph ORCH["ORCHESTRATOR - bounded state machine, typed evidence ledger"]
    direction LR
    O1["Intake"] --> O2["Recall"] --> O3["Advocates x3"] --> O4["Judge"] --> O5["Plan"] --> O6["Gate"] --> O7["Act"] --> O8["Learn"]
  end

  subgraph ENG["DETERMINISTIC ENGINES - no LLM inside"]
    direction LR
    G1["Signature library<br/>prosecution + defence"]
    G2["Contagion + rarity<br/>personalised PageRank"]
    G3["Calibrated judge<br/>log-odds ledger"]
    G4["Policy engine<br/>rules R1 to R10"]
    G5["Response simulator<br/>denies / confirms / silent"]
  end

  subgraph TG["TIGERGRAPH via MCP server"]
    direction LR
    T1[("Transaction graph")]
    T2[("Case memory")]
    T3[("Vector indexes")]
    T4[("Installed GSQL<br/>every query takes as_of")]
  end

  subgraph OFF["OFFLINE - build time"]
    direction LR
    F1["Loader<br/>708 MB CSV to typed graph"]
    F2["Time-machine backtest"]
    F3["Calibration tables<br/>JSON, no ML at runtime"]
  end

  LLM["LLM - 3 schema-checked calls per case<br/>plan evidence, explain, write SAR"]

  ENTRY --> ORCH
  ORCH --> ENG
  ENG --> TG
  OFF -. "loads + calibrates".-> TG
  F3 -. "coefficients".-> G3
  ORCH -. "phrasing only".-> LLM

  classDef entry fill:#F3F4F6,stroke:#6B7280,color:#111827;
  classDef agent fill:#EEF0FF,stroke:#5B5BD6,color:#1B1B4B;
  classDef tgc fill:#E6FAF6,stroke:#0F9D8A,color:#063D36;
  classDef control fill:#FFF4DC,stroke:#D98E04,color:#4A3000;
  class E1,E2,E3,E4 entry;
  class O1,O2,O3,O4,O5,O6,O7,O8,LLM agent;
  class T1,T2,T3,T4,F1,F2,F3,G1,G2 tgc;
  class G3,G4,G5 control;

indigo = agent and judge, teal = graph and graph-derived work, amber = control and side effects.

Five principles behind it

#	Principle	Why it matters
1	The LLM only phrases	It cannot add an action, change a route, set exposure, or invent an ID. That is what makes a mid-sized open model safe here.
2	Read through MCP, act through policy	One loggable tool contract, and no side effect without a rule citation.
3	Time is enforced in the data layer	The alert's `opened_at` is 1 to 6 hours after the flagged transaction, and the file runs to Dec 31. Every query takes `as_of`, and a leakage test fails if any row is later than the trigger.
4	Graceful degradation	Every LLM reply is validated, retried twice with the validator's error fed back, then replaced by a deterministic template. A valid case file is always written.
5	The graph is the source of truth	The answer JSON is rendered from the graph and checked: IDs exist, exposure equals the sum of affected transactions, `sar.file` matches `FILE_REPORT`.

How TigerGraph is used

TigerGraph is the substrate for the reasoning, not a place to park rows. The schema is shaped around the questions the agent asks, so each question becomes a one-hop lookup.

flowchart LR
  subgraph TXN["Transaction graph"]
    Customer -->|OWNS| Card
    Card -->|MADE| Transaction
    Transaction -->|NEXT| NextTxn["Next txn on card"]
    Transaction -->|FROM_DEVICE| DeviceProfile["DeviceProfile<br/>carries rarity weight"]
    Transaction -->|BILLED_IN| BillingRegion
    Transaction -->|PURCHASER_EMAIL| EmailDomain
    Transaction -->|IN_EPISODE| Episode["Episode<br/>derived burst window"]
    Card -->|MEMBER_OF| Ring["Ring<br/>derived component"]
  end

  subgraph MEM["Case memory"]
    ClosedCase -->|MATCHES| Pattern
    Case -->|MATCHES| Pattern
    Case -->|RETRIEVED| ClosedCase
    Case -->|HAS_EVIDENCE| Evidence
    Case -->|RECOMMENDS| Action
    Action -->|UNDER_RULE| PolicyChunk
    PolicyChunk -->|DESCRIBES| Pattern
  end

  ClosedCase -->|INVOLVES| Transaction
  Case -->|INVOLVES| Transaction
  Case -->|CONNECTED_TO| Card
  Case -->|SUSPECTS| Ring
  Evidence -->|CITES| Transaction

  classDef derived fill:#FFF4DC,stroke:#D98E04,color:#4A3000;
  class Episode,Ring,DeviceProfile derived;

Amber vertices are the ones I added beyond the suggested schema. They are built offline so the agent's questions stay local to a card's neighbourhood.

Four ways the graph does the heavy lifting

Rarity-weighted devices. Each DeviceProfile stores how many customers use it, its share of New marks, its share behind an anonymous proxy, and an IDF weight. A generic Windows/Chrome profile shared by 1,000 customers gets a near-zero weight. The ring fingerprint gets a high one.

My first ring rule flagged 212 of 9,706 device profiles, mostly common iPhones. Requiring 5 to 200 users, New on 90% of uses, and an anonymous proxy on 80% left the real ring and very few others.

A score-blind contagion sweep. A personalised PageRank starts from confirmed-fraud entities and runs over the card, device, region and email graph. It ranks unflagged cards by proximity to known fraud and ignores the bank score. One query on the ring fingerprint recovers victims the model never flagged.

Case memory as GraphRAG. Closed cases attach to the same transactions, cards and devices that live traffic touches, and recall works three ways:

Recall mode	What it does
Structural	Closed cases on the same device or card neighbourhood, ranked by overlap, rarity and recency
Semantic	Vector search over case notes, policy and regulatory text
Numeric	Nearest neighbours in a 32-dimension PCA of the Vesta features

The three results are compiled into one brief for the LLM: what we know, what conflicts, what is missing.

Memory uses shape matches only, never "same customer". Closed history is 84% fraud, so anyone with history would look guilty.

Causal write-back. Every new case becomes a Case vertex linked to its transactions, cards, ring and pattern, with retrieved cases as RETRIEVED edges. Cases are processed in opened_at order, so case N may recall earlier cases and never a later one.

The agentic capabilities

The investigation loop

flowchart TD
  T["Trigger"] --> I["1. Intake<br/>pin as_of, resolve card"]
  I --> M["2. Recall<br/>prior cases, 3 ways"]
  M --> PR["Prosecution"]
  M --> DE["Defence"]
  M --> SW["Sweep"]
  PR --> J["Judge<br/>p, confidence, evidence classes"]
  DE --> J
  SW --> J
  J --> S{"Stop rule met?<br/>p >= 0.85 or p <= 0.15<br/>with 2 independent classes"}
  S -- "yes"--> A["Policy engine<br/>choose actions + routes"]
  S -- "no, budget left"--> V["Plan<br/>best value-of-information request"]
  V --> R["Reply<br/>customer / step-up / analyst"]
  R --> J
  S -- "no, nothing left to try"--> E["Escalate to analyst<br/>with gap list"]
  E --> A
  A --> X["Explain, SAR, write case to graph"]

  classDef agent fill:#EEF0FF,stroke:#5B5BD6,color:#1B1B4B;
  classDef tgc fill:#E6FAF6,stroke:#0F9D8A,color:#063D36;
  classDef control fill:#FFF4DC,stroke:#D98E04,color:#4A3000;
  class PR,DE,J,V,R agent;
  class SW,M tgc;
  class A,E,X,S control;

Budgets: 3 evidence rounds, about 20 graph calls, 1 request of each type.

1. Adversarial advocates

Prosecution and Defence run in parallel over the same as-of graph. An unanswered defence test is reported as uncertainty, never ignored. The bank score is not a signature. It enters as one evidence class through a learned, non-monotonic curve.

The meaning of a signal flips by zone. At score 0.8+, 98% of cleared online alerts show a New device, against 36% of fraud. So "New device" is defence evidence in that zone and prosecution evidence below it.

Prosecution signatures	Defence signatures
Test then spend	Trip continuity (a clone would keep spending at home)
Off-profile burst	Device succession (new phone replaces old, same brand and OS)
New-device attribution	Recurring cadence, with a coincidence test
Out-of-region while home continues	Baseline-consistent spend
Threshold hugging (structuring)	Hub hygiene (down-weight aggregate customers)
Shared rare fingerprint	Novelty detector (flags patterns nothing explains)

2. A calibrated judge, not a vibe

Fraud probability is graded for calibration and drives every policy threshold, so it is computed, not guessed:

prior (reset for 84% fraud history) → + each evidence item's log-likelihood ratio → same class counts once (max, not sum) → global damping tuned on backtest → fraud_probability

Evidence is tagged by class (bank score, card behaviour, device, geography, network, memory, lookalike, customer reply). Correlated evidence inside a class does not stack, so the policy's "at least two independent pieces" rule is computed, not asserted.

3. Value-of-information planning

If the stop rule isn't met, the agent scores candidate requests (customer validation, step-up auth, analyst info, one more graph hop) by expected shift in p, minus friction. It must state which two hypotheses the request is meant to separate, and that sentence becomes the recorded "why more evidence was requested".

4. Response-conditional planning

Replies aren't provided, so before asking, the agent precomputes the plan for all three outcomes. The simulated reply is drawn from its own pre-reply posterior, and the assumption is recorded.

flowchart TD
  ASK["Ask the customer"] --> D["Denies"]
  ASK --> C["Confirms"]
  ASK --> Z["Silent for 24h"]
  D --> DR["Rule R2<br/>BLOCK_CARD, CREATE_CASE,<br/>FILE_REPORT if threshold met,<br/>MONITOR_CONNECTED_CARDS"]
  C --> CR["Rule R3<br/>CLOSE_NO_FRAUD"]
  Z --> ZR["Rule R4<br/>MONITOR_CARD, DECLINE_TRANSACTION,<br/>escalate if exposure over $500"]
  classDef bad fill:#FDECEC,stroke:#D1242F,color:#4A0A0F;
  classDef good fill:#E7F8EC,stroke:#2EA043,color:#0B3D1A;
  classDef mid fill:#FFF4DC,stroke:#D98E04,color:#4A3000;
  class D,DR bad;
  class C,CR good;
  class Z,ZR mid;

All three branches are stored as a contingency object, so the before-and-after recommendation is a lookup, not an improvisation, and an analyst can override the simulated reply live.

5. A policy engine that can't be sweet-talked

Rules R1 to R10 are a decision table in code. Exposure is computed from transaction IDs, never by the LLM. The executor only runs auto actions:

Route	Actions
Auto (executed)	`CREATE_CASE`, `MONITOR_CARD`, `VERIFY_WITH_CUSTOMER`, `WARN_CUSTOMER`, `ESCALATE_TO_ANALYST`, `CLOSE_NO_FRAUD`
L1 team lead	`DECLINE_TRANSACTION`, `BLOCK_CARD` if exposure up to $2,500
L2 fraud manager	`BLOCK_CARD` above $2,500, `BLOCK_ALL_CARDS`, `FILE_REPORT`

L1 and L2 actions appear as approval cards in the console, each showing the rule that required it.

6. A time machine

Any closed case can be replayed as if it had just arrived, with the graph frozen at its as_of. That is how thresholds and likelihood ratios are calibrated, and how the UI can show a live calibration plot.

A live case, step by step (TST-003)

The screenshot at the top is this exact run. An analyst wrote: "several cards this month show purchases from the same unusual device profile."

sequenceDiagram
  autonumber
  participant A as Analyst
  participant AG as Agent
  participant TG as TigerGraph (MCP)
  participant PE as Policy engine
  A->>AG: Review $74.96 online purchase on card T9003-K1
  AG->>TG: Card history, as_of = alert time
  TG-->>AG: 47 earlier payments
  AG->>TG: Run fraud signatures (prosecution)
  TG-->>AG: 2 signs: device is New behind anonymous proxy, shared by 12 customers
  AG->>TG: Sweep from the shared device profile
  TG-->>AG: 5 other cards still live on it, 6 already handled
  AG->>TG: Run innocence signatures (defence)
  TG-->>AG: 1 reason it could be innocent: amount within card p95 ($97.20)
  AG->>PE: p = 92%, classes: device, network, geography
  PE-->>AG: Likely fraud, $187.33 at risk across 2 payments
  AG-->>A: Verdict, reasoning with "Why" lines, customer-friendly message

Step	What the agent did	Why it matters
1. Read the alert	Found 47 earlier payments, used only what was known at alert time	No look-ahead
2. Look for fraud	2 signs, including a rare New device behind an anonymous proxy used by 12 customers	Graph-derived evidence, not the score
3. Look for innocence	Found 1 reason: amount is normal for this card	The defence gets a real hearing
4. Verdict	92%, very likely fraud, $187.33 at risk across 2 payments	Computed by the judge and policy engine, not by the LLM
5. Explain	Customer-friendly message, next steps	Plain language, not a risk-model dump

Each step has an italic "Why" line, and a "slow down so I can watch" toggle lets a human follow the reasoning live.

What makes this different

Typical fraud agent	Fraud Investigator
Trusts the bank score	Treats the score as one witness on a learned, non-monotonic curve
Hunts only for fraud signals	*Prosecution and* defence** in parallel
"Same customer" memory	Shape-matched memory, aware that customers are aggregates
LLM decides the action	LLM only phrases; a policy engine decides
Probability from vibes	Log-odds ledger with independence classes
Sees the future by accident	`as_of` enforced in the data layer, with a leakage test
Only finds flagged cards	Score-blind sweep finds the ring at score 0.05

What I learned

Profile before you architect. The most valuable hour was measuring the data. Adversarial advocates, a non-monotonic score and a score-blind sweep all came from three surprising charts.
"Explainable" needs structure, not prose. Forcing every claim to cite entity IDs, and rejecting any file whose IDs don't exist in the graph, cut hallucination far more than prompting did.
Constrain the model and it gets safer, not weaker. Three schema-checked phrasing calls with template fallbacks meant a smaller model was enough.
Label weak evidence honestly. Anonymous Vesta features carry real signal (AUC 0.845 alone), but they appear as "engineered features, unnamed", never as an explained reason.
Simulation can flatter you. An assumed customer denial pushed a weak 0.65 to 0.97. A system that lets its own simulated evidence inflate confidence is grading itself, so the fix is to report the pre-reply probability and let only the actions reflect the assumption.
Measure specificity on the whole population. My first ring rule looked great on the target cases and terrible across 9,706 device profiles.

What I'd improve with more time

Area	Status	Next step
Full time-machine backtest	Harness and replay built	Publish per-pattern accuracy, per-signature precision and the calibration curve `[insert results]`
Account-takeover signature	Not built	Add to the prosecution set
Numeric lookalike channel	Not built	PCA-32 vectors and kNN restricted to labelled exemplars
Full PageRank sweep	One-hop version today	Run personalised PageRank in-database with the GDS library
Vector search in TigerGraph	Planned	Move semantic recall in-database
Monitor mode	Optional	Let the agent raise alerts on its own from graph contagion
Learn from analysts	Candidate patterns proposed	Feed analyst-named patterns back into signatures and calibration
Narration model comparison	Planned	Compare models on schema pass rate, citation accuracy and SAR completeness

The verdict

A fraud agent should not be a persuasive storyteller. It should be a disciplined investigator that argues both sides, states its uncertainty, cites everything, and leaves the final decision to auditable policy. TigerGraph is what makes that possible: it holds the transactions, the memory and the time-bounded queries in one place, and the agent reaches all of it through a single MCP tool surface.

Stack

TigerGraph · TigerGraph MCP server · GSQL · Ollama (gemma4:31b-cloud, nomic-embed-text) · Next.js + TypeScript · Zod · DuckDB · Python

DEV Community: Pranav S Salian