Every e-commerce fraud team owns a corpus nobody thinks of as a corpus: chargeback case notes, analyst annotations, rule descriptions, dispute outcomes. Years of institutional memory about how fraud actually happened at your shop. And when an analyst needs it — "have we seen this pattern before?" — the search box on top of it is usually one of two half-solutions.
The librarian (sparse retrieval, BM25): exact keyword matching. Ask for "freight forwarder" and she finds every case note containing those words. Fast, predictable — and blind to the case written up as "reshipping mule address."
The scholar (dense retrieval, embeddings): understands meaning. Ask about "freight forwarder" and he also hands you the reshipping-mule cases, because in vector space they're neighbors. But ask him for rule VEL-013 and he returns cases that feel similar — while the one case literally tagged VEL-013 ranks fourth.
Fraud ops queries are brutally bimodal: half are exact identifiers (rule IDs, decline codes, BIN ranges, SKUs), half are fuzzy patterns ("expensive electronics, rushed shipping, new account"). Each retriever fails on exactly the half the other one wins.
The sample data
A miniature version of the corpus every fraud team has:
cases = [
"CB-1041: stolen card, high-velocity checkout, billing/shipping ZIP mismatch, "
"expedited shipping to freight forwarder in Miami. Rule VEL-013 fired late.",
"CB-1102: account takeover via credential stuffing, saved card used for "
"three gift card purchases within 9 minutes. Customer reported lockout.",
"CB-1177: friendly fraud — legitimate customer disputed electronics order "
"after delivery confirmation. Photo proof of delivery, dispute lost anyway.",
"CB-1203: new account, high-value order, mismatched device fingerprint, "
"reshipping mule address flagged by manual review. Order cancelled.",
"CB-1250: card testing pattern — 42 declined $1 authorizations from one IP "
"range, then a successful $890 purchase. Decline code 4863 repeated.",
"CB-1288: promo code abuse, 15 accounts created same day redeeming "
"FIRSTBUY20, all shipping to two addresses in the same building.",
]
The bimodal failure, executed
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("all-MiniLM-L6-v2")
E = model.encode(cases, normalize_embeddings=True)
def dense_search(query, k=3):
q = model.encode([query], normalize_embeddings=True)
scores = (E @ q.T).ravel()
return sorted(zip(scores, cases), reverse=True)[:k]
# Query type 1: fuzzy pattern — the scholar's home turf
for s, c in dense_search("new customer buying expensive items shipped somewhere suspicious"):
print(f"[{s:.3f}] {c[:70]}")
[0.612] CB-1203: new account, high-value order, mismatched device fingerprint,
[0.548] CB-1041: stolen card, high-velocity checkout, billing/shipping ZIP mis
[0.401] CB-1177: friendly fraud — legitimate customer disputed electronics or
No keyword overlap with "reshipping mule address" — the embedding found it anyway. Now the other half:
# Query type 2: exact identifier — where the scholar embarrasses himself
for s, c in dense_search("VEL-013"):
print(f"[{s:.3f}] {c[:70]}")
[0.229] CB-1250: card testing pattern — 42 declined $1 authorizations from on
[0.204] CB-1102: account takeover via credential stuffing, saved card used fo
[0.191] CB-1041: stolen card, high-velocity checkout, billing/shipping ZIP mis
The one case that literally contains VEL-013 ranks third, barely above noise. A rule ID has no semantics for the embedding — it's just a rare token. Rare tokens are precisely what BM25 was built to reward: run the same query through BM25 (pip install rank-bm25) and CB-1041 is first by a mile.
The fix is a fusion, not a choice
from rank_bm25 import BM25Okapi
bm25 = BM25Okapi([c.lower().split() for c in cases])
def hybrid(query, k=3, rrf_k=60):
"""Reciprocal rank fusion: rank-based, so BM25's 0-15 scale and
cosine's 0-1 scale never need to be reconciled."""
dense_rank = {c: r for r, (_, c) in enumerate(dense_search(query, k=len(cases)))}
sparse_order = np.argsort(bm25.get_scores(query.lower().split()))[::-1]
sparse_rank = {cases[i]: r for r, i in enumerate(sparse_order)}
fused = {c: 1/(rrf_k + dense_rank[c] + 1) + 1/(rrf_k + sparse_rank[c] + 1)
for c in cases}
return sorted(fused.items(), key=lambda x: -x[1])[:k]
for q in ["VEL-013", "suspicious reshipping pattern new account"]:
print(f"\nquery: {q!r}")
for c, s in hybrid(q):
print(f" [{s:.4f}] {c[:65]}")
query: 'VEL-013'
[0.0323] CB-1041: stolen card, high-velocity checkout, billing/shippi
[0.0164] CB-1250: card testing pattern — 42 declined $1 authorization
[0.0161] CB-1102: account takeover via credential stuffing, saved car
query: 'suspicious reshipping pattern new account'
[0.0325] CB-1203: new account, high-value order, mismatched device fi
[0.0316] CB-1041: stolen card, high-velocity checkout, billing/shippi
[0.0163] CB-1288: promo code abuse, 15 accounts created same day rede
Both query types now land their correct top hit. The fusion is ~10 lines. Industry benchmarks put hybrid at 15-30% better recall than either method alone — and in a fraud queue, recall is analyst minutes and missed patterns, not an abstract metric.
The resolution is the usual one: this isn't a model problem, it's a data problem. Your case notes are a dataset; the retrieval layer is a pipeline over it. Treat it with the same rigor as your feature store and the "have we seen this before?" question starts getting answered in seconds.
The principle: keyword search and semantic search aren't competitors. In a fraud stack, they're each other's error correction.
Which one is your case-search missing today — the librarian or the scholar?
I'm Vinicius Fagundes — principal data engineer and MBA lecturer in São Paulo. I build fraud and risk analytics pipelines for e-commerce through vf-insights.com.
Top comments (0)