I tried to beat the simplest baseline with a fraud trust score. Pre-registering the test taught me this.

Otu Jacob — Sun, 25 Oct 2026 08:26:01 +0000

Evidence in this post comes from synthetic data only. Nothing here shows how the system performs on real fraud.

A fraud model gives an analyst a score. It does not say how far to believe that score for this particular case. I have been building a decision-support system called Assay around that gap. Next to each risk score it produces a Trust Index from 0 to 100, meant to say how much a person can lean on the recommendation. A High-trust case can be waved through faster. A Low-trust case goes to a senior analyst.

A score like this is easy to build and hard to justify. Any model can emit a number between 0 and 100. The real question is whether the number tells you anything the model's own confidence didn't already tell you. This post is about how I tested that, and what I found when I wouldn't let myself move the goalposts.

The setup

The Trust Index combines six components:

model confidence
cohort reliability (how often the model was right on similar past cases)
explanation reliability (are the SHAP attributions stable under small perturbations?)
familiarity (does this case look like anything the model has seen?)
data drift
data quality

The detection side is an ensemble of XGBoost, a random forest and an isolation forest, with calibrated probabilities. The system is recommend-only. It never acts on its own.

Figure 1. How a recommendation is produced. The Trust Index runs beside the risk score, and the policy engine uses both.

The test I cared about was simple. Among the model's wrong recommendations, does the Trust Index rank them lower than correct ones? I measured that as AUROC over "was this recommendation wrong?", and compared it with three baselines of increasing strength:

B1: distance of the score from the decision threshold
B2: disagreement between ensemble members
B3: the model's maximum class probability

B3 is the one that matters. It costs nothing, and a lot of elaborate uncertainty work fails to beat it.

Rule one: write the decision rule before the run

All the data is synthetic, generated with known ground truth. That lets me inject fraud types, drift and label delay, and know exactly which recommendations were wrong. It also means the data can only show that the machinery works, not that the idea works on real fraud.

My first two runs were small, and ended inconclusive. The Trust Index was ahead of B3 on both seeds, but there were only 47 and 57 errors per run, so the interval included zero.

The tempting move is to keep running until a number looks good. Instead I wrote the decision rule into the docstring of the validation script before running anything new:

Supported only if the mean over seeds of AUROC(Trust Index) minus AUROC(B3) has a 95% interval that excludes zero on the positive side.

Then I ran twelve fresh seeds that had never been used for a design decision, about 1,400 errors in total. The unit of analysis was the seed, not the case, because cases within one seed share a generator.

Figure 2. The validation workflow. The rule is fixed before the run and applied as written.

Trust Index minus	Mean AUROC difference	95% interval over seeds	Ahead in
B1, distance from threshold	+0.124	+0.090 to +0.157	12 of 12
B2, ensemble disagreement	+0.080	+0.050 to +0.109	12 of 12
B3, max class probability	+0.030	+0.009 to +0.051	9 of 12

Figure 3. Trust Index AUROC minus each baseline, mean over twelve seeds with 95% intervals.

By the rule I'd set, it passes. But a lower bound of +0.009 on a base AUROC of 0.91 is a weak result, and I report it as one. It is ahead in 9 of 12 seeds, a sign test over seeds gives p = 0.15, and all twelve seeds share one generator, so they show stability over sampling, not over kinds of fraud.

Rule two: ablate your own components

Pre-registration also covers the parts. For each component I recomputed the index without it and measured the change in AUROC:

Component removed	Change in AUROC	95% interval	Reading
cohort reliability	-0.035	-0.045 to -0.025	large effect, 12 of 12 seeds
model confidence	-0.009	-0.014 to -0.005	small effect, 12 of 12 seeds
explanation reliability	+0.002	-0.001 to +0.005	no measurable value
familiarity	+0.009	+0.000 to +0.017	removing it helps in 10 of 12 seeds
drift, data quality	0	0	nothing, by construction

Figure 4. Change in AUROC when each component is removed. Left of zero means the component helps.

The honest reading is uncomfortable. Almost all of the discriminating power comes from one component, cohort reliability, which depends on matured outcomes. A new deployment won't have them at first. Explanation reliability is the most expensive component to compute (about 27 ms per case) and showed no value on any run. Drift and data-quality components show nothing because the synthetic data is clean and stationary. Stress tests show their mechanisms work, but that is a different claim.

I did not keep the component because it was elegant. It stays in the report as a product decision, to be made on data separate from this.

Rule three: let the pre-set rule say no

I also built a calibrated mode, where a score of 80 means "right about 80% of the time". The calibration worked, with an expected calibration error of 0.005, against 0.268 for the raw score read as a probability. But the same pre-set rule asked for ranking quality at least as good as the provisional score, and it was not shown to be. The calibrated variant ranked errors slightly worse on the stable part of each window, with an interval reaching below the limit I'd set.

By the rule, the verdict was to stay off. I did not loosen the limit because the calibration number was impressive.

A second variant kept the provisional order and learned only a monotone map to a probability. It passed its test (the ranking is identical by design, ECE 0.0046). But the system's own promotion gate, which the server enforces in code, still passes on only 2 of 12 seeds, mainly because beating B3 with an interval that excludes zero is hard on a single window. So calibrated mode stays off by default, and the server refuses to run it for a tenant whose bundle did not pass the gate on that tenant's own outcomes.

What broke along the way

Two things I'd flag for anyone doing similar work.

Method changes after seeing results. I made five. For example, a plain k-NN distance percentile treated every sparse region as novel, including known fraud such as account takeover, so I switched to a local distance ratio. I removed the isolation forest from the familiarity component, because it measures rarity, not novelty. Each change was made in response to a stress-test failure and then checked on a fresh seed that hadn't been used to choose it. They are listed in the validation report so nobody reads the result as if the design had been fixed in advance.

An experiment that could not answer anything. My first design for the calibrated-mode test used a short test window. A smoke run showed only 10 of 145 errors fell in the oldest 45% of the window, because nearly all errors arrive after the novel fraud type appears. The model correctly refused to fit. I changed the dataset to a longer window before running the real seeds. Nothing about the real seeds informed that choice.

What I'd tell someone building something similar

Compare against the cheapest baseline first. B3 cost nothing and was the hardest to beat.
Write the decision rule down before the run. It turns "does this work?" into a question with a yes or no answer, and it stops you relaxing the bar when you like the result.
Treat the seed, not the case, as the unit of analysis when cases share a generator.
Ablate everything. The component I was proudest of had no measurable value, and the one I'd treated as plumbing carried most of the signal.
Report the negative results in the same document as the positive ones.

Limits

All data and analysts are synthetic. Passing a test here shows the machinery works, not that fraud looks like my generator.
Selective-label bias is not modelled: every synthetic case gets an outcome. Real fraud outcomes arrive late, and only for cases someone investigated.
Intervals are wide because errors are few per run.
The twelve seeds share one generator and one configuration.

The next step is a pilot with matured outcomes from a real institution. That is the only thing that can change these verdicts, and I'd rather say so than claim more than I've shown.

About the author

I'm Otu Samuel Jacob, a product manager working where payments, fraud and machine learning meet. Over seven years in regulated financial institutions in Nigeria & the UK I owned products at scale: a mobile banking payments capability that processed NGN600B+ in its first year, a POS acquiring portfolio of 25,000+ terminals, and an BaaS launch for institutional clients. I partnered with risk and compliance teams to deploy fraud-monitoring controls.

I completed an MSc in Data Science and Big Data Analytics with Distinction at Wrexham University in September 2026, and I work hands-on in Python and SQL. I designed and validated Assay, a recommend-only fraud decision-support system with a calibrated risk score, a Trust Index, and analyst review at every decision. I write about testing ML systems honestly: pre-set decision rules, fresh seeds, and reporting negative results.

Questions and challenges to the method are welcome in the comments.

DEV Community: Otu Jacob