DEV Community: Oluwagbade Odimayo

The fraud that pays in full

Oluwagbade Odimayo — Sun, 12 Jul 2026 19:21:19 +0000

Two credit accounts land on a risk analyst's desk in the same month. Both sit near their respective limits. Both have stopped paying. Both are sliding into delinquency. On this month's statement, they are indistinguishable.

One is a family that lost an income and is drowning. The other is a fraudster who spent half a year building a spotless record so the line would be raised, and has now drawn every available dollar with no intention of paying it back.

Freeze the wrong one and you either let a fraud loss run to charge-off, or you slam the line on a struggling borrower at the worst possible moment. This is the problem behind [bustout detection (https://github.com/gbadedata/bustout-detection), and it is the latest in a short series about fraud signals that are not what they first appear. The first was a card-fraud ring that turned out to be an artefact of the data. The second (https://github.com/gbadedata/mule-network-detection) was a money mule the graph flagged, and the evidence did not support. This one is the opposite trap: the fraud that hides by looking like your best customer, and the struggling borrower a lazy model would punish in its place.

What a bust-out is

Bust-out is first-party fraud, and it is the expensive kind. There is no stolen card and no victim to call. The account holder is the fraud. The pattern runs in three acts.

First, cultivation. The account is opened and used carefully. Balances stay low, statements are paid in full, and the behaviour is so clean that the lender does what lenders do with good customers: it raises the limit, again and again. For several months this account is a model borrower, and on paper one of the best on the book.

Then the ramp. Over a few months utilisation climbs, payments shift off full and toward the minimum, and the balance grows. Often a fresh limit increase is taken right at the turn.

Then the bust. The whole line is drawn at once, cash is pulled where it can be, the payments stop, and the account rolls to charge-off and goes silent.

A model that scores an account on this month's numbers never sees it coming, because for most of the account's life there is nothing to see. The signal is not in any single statement. It is in the shape of the months.

The trap: utilisation lies

The obvious way to rank accounts for review is by how stretched they are: utilisation, or days past due. It is intuitive, and it is wrong, because genuine distress produces the same readings. I built the project on a synthetic account-month panel with four kinds of account, because no lender publishes labelled bust-out data and the whole difficulty is telling the classes apart. Alongside good accounts and steady revolvers, the panel carries genuine distress: a household whose utilisation climbs slowly, whose payments taper toward the minimum, whose spend falls as the room runs out, and who ends up delinquent. Distress and bust-out arrive at the same place. They travel there by different roads.

Here is the same set of accounts averaged over the months that matter, a bust-out in its ramp against a distressed account in trouble:

Signal	Bust-out ramp	Genuine distress
Utilisation	0.73	0.95
Utilisation slope, 3 months	0.48	0.28
Recent full-payment streak	0.76	0.00
Minimum-payment streak	0.38	2.16
Cash-advance share	0.21	0.00
Limit growth since open	1.82x	1.00x
Months since last limit increase	3.1	15.9

Utilisation, the obvious number, is higher for the honest account. Everything about the path is the opposite. The bust-out climbs faster, off a recent run of full payments that has just broken, draws cash, and has grown its limit quickly and recently. Distress has paid the minimum for a long time, draws nothing, and its limit has not moved. Rank on the level and the queue fills with the wrong accounts: on the panel, roughly seven in ten of the accounts a utilisation-ranked queue flags are genuine distress, not fraud, and a delinquency-ranked queue is worse.

Reading the path instead of the point

The fix is to stop scoring the snapshot and score the trajectory. The features that matter describe how an account is moving, not where it happens to be:

How fast utilisation is climbing, and how far it has jumped above the account's own prior peak. A bust-out ramps hard off a low base; distress creeps.
Whether a run of full payments has just broken. Bust-outs pay in full through cultivation and then stop abruptly; a distressed account has paid the minimum for a long time.
The share of activity that is a cash draw, which spikes in a bust-out and barely moves in distress.
Whether spend is accelerating or falling. A bust-out spends into the ramp; distress spends less as the room disappears.
How fast and how recently the credit line has grown, and where utilisation sat around the last increase.

A trajectory model is easy to get wrong. If a feature for a given month quietly uses a later statement, the model looks brilliant in a notebook and falls over the day it runs on live accounts that have no future yet. So the guard is a test: features computed on a truncated history must equal features computed on the full history, for the same rows.

def test_features_use_no_future_info():
    full, cols = build_features(panel)
    part, _ = build_features(panel[panel["month_index"] <= 10])
    merged = full.merge(part[key + cols], on=key, suffixes=("", "_p"))
    for c in cols:
        assert np.allclose(merged[c], merged[f"{c}_p"], equal_nan=True)

Every feature is built strictly from an account's own past, and the forward label, does the account bust within the next few months, is the only thing allowed to look ahead.

Turning it into something a team can use

A score is not a tool. The project wraps the model in the decision a risk team makes each month: monitor, reduce the line, or freeze it. The queue is ranked not by score alone but by exposure at risk, the undrawn portion of the line that a bust-out would draw at the max-out, because the point of acting early is to cut that exposure before it is taken. Each flag carries plain reasons drawn from the trajectory, and the one that separates fraud from hardship is a clean payment history that has just broken.

Evaluated out of time, training on earlier months and scoring later ones, three things stand out. The model ranks the rare bust-out label well above the point-in-time baselines: average precision around 0.96 on the synthetic test, against about 0.48 for a utilisation ranking and 0.05 for delinquency.

It is early enough to matter. At the freeze threshold the model catches most bust-outs a median of two months before the draw, while the exposure is still undrawn and a freeze still prevents the loss.

And it leaves struggling borrowers alone. This is the result that matters. In the top slice of its queue, the trajectory model's share of genuine-distress accounts is zero, against roughly seventy percent for the utilisation queue and more for the delinquency queue. Same review budget, opposite outcome. That gap is the entire argument for modelling the path.

What this is, and what it is not

Every number here comes from a synthetic panel, and it is worth being plain about what that does and does not buy. There is no public bust-out dataset, so the generator has to carry the realism, and I built it so that distress and bust-out share an ending and differ in their path, rather than being trivially separable. The exact figures move a little with library versions, and the ramp on synthetic data is cleaner than a real fraudster would leave it. Real bust-out is adversarial and shifts as controls tighten, and a live model would need recalibrating on real outcomes.

What carries over is the shape of the problem and the shape of the answer. When two very different truths produce the same snapshot, the snapshot is not where the answer lives. Rank on the level and the queue fills with the wrong accounts. Rank on the trajectory and it fills with the right ones. The best customer on the book is sometimes the one to worry about, and the way to tell is to watch how it got there.

The code, the synthetic panel, the model, the freeze queue, and the investigation queries are all in the repository.

Your fraud model doesn't know what a truck is

Oluwagbade Odimayo — Sun, 12 Jul 2026 19:06:53 +0000

A fuel card is swiped at 2:14 in the morning, at a truck stop 2,400 miles from where the same card bought diesel the day before, for 300 gallons at a single pump. Fraud?

Two of those three facts are ordinary. A long-haul driver keeps odd hours and covers ground; 2,400 miles in a day is a team or a relay rather than an impossibility, and 300 gallons is a normal fill for a rig with two saddle tanks. The third fact, a single pump putting 300 gallons into a tank that holds 240, is the one that matters. A generic card-fraud model cannot pull the three apart, because it does not know what a truck is. It sees an unusual place, an unusual hour, and a large amount, and it either shrugs or flags them all.

This project, fuel-card-fraud-monitoring, is about the part a generic fraud model misses: the semantics of fuel. It is part of a short series about fraud signals that are not what they first appear, after a card-fraud ring that turned out to be an artefact of the data and a money mule the evidence did not support (https://github.com/gbadedata/mule-network-detection).

This one carries two traps, and they pull in opposite directions. One is the driver who looks guilty and is not. The other is the fraud that looks innocent and is not.

Fuel is not retail

Retail card-fraud models live in a world of merchants, amounts, and velocities, and they are good at it. A fuel card lives somewhere else, under physical constraints a retail model never learns:

A tank has a size. A 240-gallon fill into a 240-gallon tank is fine. A 400-gallon fill is fuel going somewhere other than the truck.
Diesel trucks burn diesel. A fuel card buying premium unleaded, or merchandise, or a cash-equivalent, is worth a second look.
Trucks get a knowable range of miles per gallon. Between two fills the odometer and the gallons imply an economy, and two miles per gallon on a full tank means the fuel did not go into this truck.
Trucks run corridors. A driver has a home base and a radius they work within, and their fuelling traces a lane across the map.

None of that is in a generic model, and all of it is in this one. The features are the physics: gallons against tank capacity, fuel-type match, implied miles per gallon from the odometer, speed between consecutive fills, distance from the card's own corridor. Each is computed strictly from what was known before the swipe, so nothing from the future leaks into
the score.

The driver who looks guilty

Here is the first trap. Suppose you flag a swipe as off-route when it is far from the card's home base. It reads as reasonable, and it is wrong, because a long-haul driver is always far from base. That is the job. Flag on absolute distance and every long-haul card lights up, the queue fills with drivers doing ordinary work, and the real off-route fraud is buried under them.

The fix is to measure deviation against the card's own history, not against an absolute:

# running centroid of the card's prior fuelling sites (strictly before this swipe) off_route = haversine(lat, lon, mean_lat_prior, mean_lon_prior)
# divide by the card's own prior roaming radius, so a driver who always ranges far
# is not mistaken for one that has suddenly gone off-route off_route_ratio = off_route / (prior_mean_off + 100)

A driver who always ranges 900 miles from their centre has a large off-route distance and a small ratio: normal for them. A driver who has always stayed within 150 miles and today sits 600 miles out has a modest distance by long-haul standards but a large ratio: that is the anomaly. The model leans on the ratio, and that is why a card that roams for a living is not punished for roaming.

Two layers, on purpose

The decision at the pump has to be instant and explainable. When a card is declined at the island, someone has to be able to say why, in words, that minute. A gradient-boosted probability is not that. So the system has two layers doing two jobs.

The first is a rules engine: a handful of deterministic checks with plain-language reasons. Impossible travel between two fills. Gallons over the tank. Wrong fuel. An economy below a floor. These fire in real time, they are auditable, and they carry the pump decision with a reason a person can read: "1,138 gallons exceeds the 300-gallon tank; implied 0.8 mpg on the fill."

But rules alone are blunt in both directions. They flag 5.4% of legitimate swipes, because a fixed threshold catches ordinary variation along with fraud, and they miss the fraud that stays under every threshold. So a model rides on top as the ranker, ordering a review queue by expected loss.

One design choice is worth stating, because the obvious move is the wrong one. You could blend the rules and the model into a single score. Doing that drags the ranking down, because the rules carry their false positives into the blend. The rules earn their place as the real-time guardrail and the explanation; the model earns its place as the ranker. Keeping them separate makes each one better.

The fraud that looks innocent

Here is the second trap, and it is the reason the model exists. Consider a swipe built to stay under every rule:

# evasive: every signal kept just under its threshold, anomalous only together
gallons  = tank * uniform(0.92, 1.12)           # near the tank, not over it
odometer = prev + gallons * uniform(3.0, 4.3)   # a low economy, but above the floor
emit(..., hour=overnight, entry="manual", amount=above_card_average)

Nothing here trips a rule. The fill is close to the tank but not over it. The economy is poor but not impossible. It is hand-keyed and overnight and above the card's usual spend, but none of those alone is fraud, and roughly one legitimate swipe in twelve is hand-keyed. Each signal sits in the safe zone. Only the combination is wrong.

The rules catch about four percent of this. The model catches close to ninety, because a gradient-boosted model reads the interaction of features that no single threshold can. That gap is the entire case for the second layer. Across the test set, by typology:

Typology	Rules alone	Model
Impossible travel	100%	100%
Tank overflow	100%	95%
Wrong or non-fuel	100%	90%
Implausible economy	100%	100%
Rapid repeat	100%	100%
Off route	92%	79%
Evasive	4%	87%

The rules own the clear typologies, which is what a real-time guardrail should do. The model owns the one that hides in combination, and it does so while flagging only 0.3% of legitimate swipes, against the rules' 5.4%.

What it adds up to

On an out-of-time test of about 7,000 swipes at a 2.6% fraud rate, the model ranks fraud far above the rules alone: average precision near 0.99 on the synthetic test, against 0.75 for the rules as a ranker and 0.03 for no-skill. Two operating views matter more than the single number.

As a review queue ranked by expected loss, working the top 2% of swipes recovers 90% of the fraud value, and the top 5% recovers all of it. A team with a fixed amount of review time spends it where the money is.

As a real-time gate, declining on a hard rule or a high model probability catches 96% of fraud while declining under 1% of legitimate swipes. Adding a step-up, a prompt for a second factor, on the middle band reaches 98% of fraud while touching 1% of legitimate traffic. Most fraud stopped at the pump, almost no honest driver inconvenienced.

What this is, and what it is not

The numbers come from a synthetic fuelling panel, and it is worth being plain about that. No fuel-card programme publishes labelled fraud, so the generator has to carry the realism: real hubs and distances, drivers with home bases and radii and tanks, corridors that fills trace across the map, and a rate of hand-keyed entry high enough that entry mode is not a giveaway.

The typologies, evasive included, are injected against that backdrop. Exact figures move a little with library versions, and real fuel fraud adapts as controls tighten in a way a static generator does not.

What carries over is the shape of it. Fuel-card fraud is a domain problem before it is a modelling problem. The useful signals are physical, and a model that does not know what a truck is will both miss the fraud that hides and punish the driver who is only doing the job.

Encode the semantics, measure each swipe against the card's own history, keep a transparent guardrail for the real-time call, and let the model rank what is left.

The code, the synthetic panel, the rules engine, the model, and the investigation queries are all in the repository.

The mule that wasn't: a graph model, synthetic data, and the accounts that broke it

Oluwagbade Odimayo — Sat, 11 Jul 2026 22:58:59 +0000

This is the second of a pair. The first was about card fraud, and its lesson was that a chart I was proud of turned out to be a mirror. It looked like my model had found fraud rings, and what it had really found was my own assumptions handed back to me. This post is about money laundering, and it opens on a number I was briefly pleased with and then spent a while trying to explain away: 0.99.

That was the precision-recall AUC of a mule-detection model on a labelled dataset, with per-typology recall between 93 and 100 percent. If you have spent any time near anti-money-laundering work, you already know that a 0.99 on that problem is not a triumph. It is a symptom. Real laundering detection does not look like that, and when your numbers say it does, the thing to distrust is your numbers.

Here is what the model was trying to do, why the 0.99 was fake, and what the real data did to it.

Why laundering is a graph problem

Laundering is rare. In the data I used it runs to about 0.18 percent of transfers, and in the wild the uncommon variants are far rarer than that. At those rates, scoring transactions one at a time is close to pointless. A classifier that flags nothing is right 99.8 percent of the time, and whatever signal a single transfer carries is drowned by the base rate.

The signal is not in the transfer. It is in the shape of the money's path. Mules move funds through networks, and those networks take recognisable forms: many accounts feeding one (fan-in), one account feeding many (fan-out), money walked hop by hop along a chain, closed cycles that return to their start, and rapid pass-through where funds land in an account and leave again within hours. Lift the problem off the individual transaction and onto the structure of the account graph, and the signal concentrates where the typologies live.

So I built the account graph, computed features over it, scored transfers with those features, and ranked accounts for an investigator to review. The dataset was IBM's synthetic anti-money-laundering set, which ships with a per-transaction laundering flag and a separate file naming the typology of each laundering attempt. No bank releases real labelled mule data, so a synthetic set is the firm ceiling on what any of this can claim, and I will come back to exactly how that ceiling bit me.

The one discipline I carried over

The card-fraud post was, at bottom, about leakage: a feature that quietly peeks at the future inflates every number downstream, and none of it survives contact with reality. So this time I built the leakage guard in first.

The features split into two kinds, and keeping them apart is the whole discipline. Streaming features describe an account's history strictly before the current transfer: how many distinct counterparties it has had, how much it has moved, its pass-through ratio, and the burst counts, meaning how many counterparties arrived in a short trailing window. These use no future information, so they are what a real-time monitor could score on. A feature computed over an account's later transfers is the graph version of lookahead: knowledge you would not have at the moment you have to decide.

The second kind, retrospective features, describe an account across the whole window for an investigator working a case after the fact. They are useful, and they are never model inputs, because letting the future into training is how you build a model that works in a notebook and nowhere else.

The guard is a test, not a comment in the code. Compute the streaming features on the first half of the data, compute them again on the full data, and the early rows have to come out identical. If a later transfer changes an earlier row, information is leaking backward, and the test fails.

full, cols = prior_features(df)
for k in range(1, len(df)):
    prefix, _ = prior_features(df.iloc[:k])
    assert np.allclose(full[cols].iloc[:k].to_numpy(),
                       prefix[cols].to_numpy())

On top of those features sat a gradient-boosted model trained on a strict time split, its transaction scores aggregated to a per-account risk, and a queue that ranked accounts by expected laundered value with plain-language reasons attached, so an alert could explain itself.

All of it passed. The leakage test was green. And the model scored 0.99.

The number that was measuring the wrong thing

I had built a schema-faithful mock early on so the whole pipeline could run without the download, and I had tuned and tested against it. On that mock the model was close to perfect. Then the real file finished downloading, I pointed the loader at it, changed nothing else, and everything fell through the floor.

	synthetic mock	real IBM data
transactions	63,000	5,080,000
laundering rate	2.4%	0.18%
transaction PR-AUC	0.99	0.037
per-typology recall	93 to 100%	27 to 51%
laundered value in top 2%	~99%	54.5%

Not 0.99. 0.037. About twenty times better than random at that base rate, which is real signal, but a different universe from what the mock had promised. The same code, the same features, the same leakage guard, and the number fell by a factor of twenty-five the instant it met data I had not generated myself.

The mock had not been a test. It had been a mirror, the same one from the first post wearing different clothes. I had built the synthetic data to contain the patterns my model looked for, and I had left out the one thing that makes the real problem hard. The 0.99 was not a measurement of the model. It was a measurement of the mock, reflecting my own design decisions back at me and calling them accuracy.

Every signal that catches a mule also describes a bank

Here is what the mock left out, and what the real data is full of: legitimate high-throughput accounts.

Payment processors. Exchanges. Settlement accounts. Aggregators. Accounts that receive from hundreds of counterparties in a day and send to thousands, moving enormous volume through the network as their ordinary, entirely legal business. My mock had none of them. Every account in it was either a normal low-volume account or a mule, so the model never had to tell a mule apart from anything except a quiet legitimate account, which is easy.

Real data does not grant that. One of my investigation queries ranks accounts by fan-in, the number of distinct senders paying into them within a short window. Here is the top of that list on the real data:

        account   peak_senders_3d   total_in   laundering_share
   70-100428660               545      638329              0.000
   70-1004286A8               328      372750              0.000
 21611-8051A3FA0                93    22231920              0.000
  4726-808FAF7C0                88     1742220              0.000

Five hundred and forty-five distinct senders in three days, and a laundering share of zero. It is completely legitimate. It is also indistinguishable, by every structural signal I had built, from a collector at the center of a fan-in ring. High burst, high fan-in, balanced pass-through: a payment processor lights up all of them, because gathering money from many sources and moving it on is what a processor does and what a mule does. The structure is the same. The intent is not, and intent is not in the graph.

That is the confound, and it is the entire difficulty of the problem. On synthetic data my structural features looked like a laundering detector. On real data they are closer to a high-activity detector, and high activity is mostly legitimate.

The queue that filled up with banks

Nowhere was this clearer than the investigation queue. I had ranked accounts by expected laundered value, which is risk multiplied by the money the account moved, the same value-first logic that worked in the card-fraud project. On real data the top of the queue was dominated by legitimate accounts moving hundreds of billions of dollars, because raw value swamps everything. Two of the top ten were laundering. The other eight were banks.

The fix came from asking what a mule does that a processor does not, and the answer is timing. A processor is busy all year. A mule is busy for a few days. Its burst is not just large, it is concentrated: a large share of everything the account ever does falls inside one short window. So I added burst concentration, the peak burst over the account's total activity, near one for an account that fired in a single spike and near zero for one that is always active, and folded it into the ranking alongside a log of the value so a handful of mega-accounts could not dominate on size alone.

Here is the top of the queue after that change. The reasons column is generated from the structural signals, so each alert says why it is there:

 concentration  laundering              reasons
         0.985           1  fans in: 56 inflows; fans out: 3623 outflows; u-turn match 1.00
         0.985           1  fans in: 539 inflows; fans out: 36075 outflows; u-turn match 1.00
         1.000           0  elevated model score without a single clear structural pattern
         1.000           1  fans in: 26 inflows in a short window

That lifted the laundering share at the top of the queue from two in ten to six in ten. The accounts it surfaces now include genuine laundering hubs with tens of thousands of transfers packed into a short burst, while the always-on processors drop away. It is a real improvement, and I want to be precise about what it is not. It is a first step, not a solution. The false positives that remain are concentrated accounts the model scored high, and concentration alone cannot separate a mule from a small legitimate account that simply had a busy week.

What the real number buys you

Stripped of the fantasy 0.99, the model still does something worth having. The precision-recall AUC of 0.037 is the wrong lens, because precision and recall on a 0.18 percent base rate punish you for the base rate. The lens that matters to a review team with a fixed number of alerts they can work is value. Rank every transfer by score, walk down the list, and by the time you have reviewed the top two percent you have recovered 54.5 percent of the laundered value and 38 percent of the laundering transfers.

That is a usable operating point, and it is the number I would put in front of an investigator, not the AUC.

Per typology the picture is even and modest, which is what real detection looks like. Recall across the labelled typologies runs from 27 percent on fan-out to 51 percent on fan-in, with cycles, stacks, scatter-gather and the rest in between.

Nothing is solved. Nothing is missed. The typologies whose activity is concentrated in time do a little better than the ones that blend into ordinary volume, which is exactly what you would predict from the confound.

Getting past this ceiling is not a matter of a better classifier on the same features. It needs peer-group baselines, so an account's throughput is judged against accounts of its own type rather than against everyone, and it needs account history, so a sudden burst on a young account counts for more than steady volume on an old one. It needs to score entities and networks, not lone transfers. That is a larger project than this one, and I would rather ship the version that is clear about its own ceiling than the one that hides behind a 0.99.

The same lesson, twice

Both of these projects ended in the same place, and I think it is the place worth ending in. In the first, the trap was temporal: a feature that peeked at the future made the model look like it had learned something it had not. This time I avoided that one. The leakage test was green from the start. And a subtler version of the same self-deception got me anyway, because a synthetic dataset is a kind of lookahead too. It lets you see a world you designed to be solvable and mistake it for the world.

The discipline that catches both is the same. When a number makes you look good, that is the moment to distrust it and go looking for the thing that breaks it. In the card-fraud project the thing that broke the chart was real transactions. Here it was a legitimate account moving a billion dollars that trips every mule signal there is. Both were sitting in the data the whole time. The only question was whether I would go find them before someone else did.

The code, the leakage test, and the full write-up of the real numbers are on GitHub.

The fraud ring that wasn't: real transaction data and the results you can trust

Oluwagbade Odimayo — Sat, 11 Jul 2026 09:07:41 +0000

A device-sharing signal looked perfect on my test data and collapsed on the real thing. So did my model's best score. Both failures pointed at the same lesson.

The alert sitting at the top of my fraud queue read: device shared by 4,661 cards. It was a legitimate purchase, and it was the best thing that happened in the whole project.

That alert was wrong because a chart I trusted was wrong, and the chart was wrong because I had, without meaning to, built the test data that made it look right. This is a story about two results that flattered me, how real data destroyed both, and why the results that survive contact with reality are the only ones worth putting your name on.

What I was building

A transaction fraud scoring pipeline, built around the constraint that actually shapes fraud operations: a review team can only look at a fixed number of alerts a day. You do not get to flag everything. So the goal is not "detect fraud" in the abstract. It is to prevent the most fraud value inside a fixed review budget, keep false declines low, and notice drift before precision quietly rots. Ranking under a budget, not raw accuracy, is the product. A model that catches most fraud but needs a human to review half of all transactions is worthless.

I built it on synthetic data, then took it to the real thing: the IEEE-CIS Fraud Detection dataset, 590,540 card-not-present transactions, a 3.5% fraud rate, and about $3.08M in fraud value. IEEE is a good stress test because it withholds what a toy pipeline leans on. There is no account id and no geolocation. You reconstruct identity from anonymised card attributes yourself, which is where the interesting work is, and where I got caught.

The signal that looked perfect

Organised fraud reuses infrastructure. One device drives many "different" cards. So a natural hypothesis: count the distinct cards seen on a single device, and a device touching a dozen cards is not a coincidence, it is a ring.

On my test data this worked beautifully. I bucketed transactions by how many cards had touched the device, and the fraud rate climbed cleanly with sharing, all the way to 100% at the top. It was the kind of chart you screenshot for a slide. Sharing on the x-axis, fraud rate marching up the y-axis, a tidy story about catching rings before the model even weighs in.

I believed it. I should not have, because I had built the data that guaranteed it. Every simulated device in my mock was unique, so the only devices touching many cards were the rings I had injected myself. The chart was real. It was also a mirror. It reflected my assumption back at me and I mistook the reflection for evidence.

The collapse

Then I ran it on the 590,000 real transactions, and the same chart came back flat. Fraud rate barely moved as sharing increased, and the biggest bucket, devices shared by seven or more cards, held nearly 18,000 transactions at only 8.6% fraud. That is a whisper above the 3.5% base. Whatever this was, it was not a ring detector.

And the queue produced that alert. Device shared by 4,661 cards, on a transaction that was not fraud. Here is the actual row, sitting near the top of the queue by expected loss:

 TransactionID  card1   amount  risk_score  expected_loss                 reason_codes  is_fraud
       3513661   6021  1600.00    0.417203     667.525150  device shared by 4661+ cards         0

The cause was obvious once I looked, and a little embarrassing. In the real data the device field is not a fingerprint. Its most common values are Windows, iOS Device, MacOS, Trident/7.0. Those are operating system and browser families, shared by enormous numbers of ordinary people. "Many cards on this device" mostly meant "many people use Windows." I was counting popularity and calling it fraud.

The part that stung: I had already seen this exact failure and guarded against it, just not here. Earlier I had ruled out "cards per email domain" and "cards per billing region" as ring signals, because everyone uses a handful of domains and lives in a handful of regions, so those counts are dominated by how common a value is, not by fraud. I knew the rule. I applied it to email and address. I missed it for the device, because my mock had quietly made every device unique, so the trap never sprang in testing.

That is the first lesson, and it is bigger than one field: a low-cardinality, high-frequency value will masquerade as a strong signal in any count of shared entities, and test data you built yourself will hide it if you built it too clean. The mock did not just fail to catch the bug. The mock made me confident in it.

Fixing it without lying

Two things needed fixing: the signal and the explanation.

For the signal, a device string alone is too coarse, so I built a more specific fingerprint by combining the device with the browser and the screen resolution, which splits one giant "Windows" bucket into many specific ones. Then, instead of only handing the model a raw shared-card count, I also gave it how common each fingerprint is overall, so the model can separate a rare fingerprint shared by five cards, which is interesting, from a popular one shared by five thousand, which is not. Both are structural counts. Neither touches the label, so neither can leak.

For the explanation, I gated the human-facing parts. The reason code and the deterministic ring rule now only fire when a fingerprint is specific enough to mean something. After that, the queue can no longer tell you a Windows machine is shared by four thousand cards. That sentence cannot be produced anymore.

The honest result: once the generic families are gated out, sharing does correlate with fraud, roughly 9% at two cards and 15% at four to six, against the 3.5% base. But it is modest and non-monotonic, and the largest specific bucket falls back toward the base rate. It is a useful investigator lens and a decent cold-start rule for the window before the model has evidence. It is not the clean 100% detector my test data promised. Demoting it from headline result to tested hypothesis was the most senior move in the project.

The same lesson, wearing a different disguise

While I was being honest about the device signal, the model handed me the same lesson in a different form.

IEEE has hundreds of engineered features, and strong solutions lean on them, so I fed the model all of them: the full Vesta set, the counting and timedelta columns, the identity fields, frequency encodings, and a client key anchored on a registration-date signal. Feature count went from 52 to about 460.

PR-AUC went from 0.444 to 0.442.

Adding the dataset's best features moved the top of the queue but not the curve. So I gave the model more capacity, deeper trees, more iterations, which took it to 0.468. More feature engineering took it to 0.468 again. Four configurations, and the ceiling sat at about 0.47.

Configuration	Features	PR-AUC
Thin baseline	52	0.444
All Vesta + timedelta + identity	451	0.442
Higher-capacity model	451	0.464
D1-anchored client key + prior-amount	459	0.468

When your best features do not move the curve and neither does more model, the ceiling is not in your features or your learner. It is in the method. The gap between 0.47 and the scores on the public leaderboard is bridged almost entirely by one move I refused to make, and it is the same trap as the mock, wearing a different disguise.

The move is aggregating features across the entire dataset. Compute the mean transaction amount for a client over all their rows, including future ones, and attach it to today's transaction. It reliably lifts the offline score. It is also feature lookahead: in production you do not have the client's future transactions when you score the current one, so a feature built from them is a promise you cannot keep. It is a flattering number you construct by peeking at data you will not have. Exactly like the mock, it produces a result that looks like evidence and is really a reflection of what you already assumed you would get. Every aggregate in my pipeline is computed strictly from prior rows, current transaction excluded. Weaker on paper, honest at decision time.

So 0.47 is not a disappointing number. It is what a single calibrated model can do on this data using only information you would actually have when the transaction arrives. I would rather ship that and explain it than post a bigger number I could not deploy or defend in a review.

What actually ships value

If the score is not the headline, the operating result is, because it maps to what the team does.

The queue ranks every transaction by expected loss, probability times amount, so it prioritises the cases that cost the most, not just the ones most likely to be fraud. At the cost-chosen operating point on held-out data, the model recovers $380,070 of $609,934 in fraud value, about 62%, at the review load the cost model picked. The value-versus-budget curve shows the rest of the trade-off: for whatever review capacity a team has, how much fraud value it recovers.

That curve is the real deliverable. It lets someone with a fixed headcount choose their point on it deliberately, instead of accepting whatever threshold a model happens to output. Probabilities are calibrated too, which is what makes multiplying a score by an amount mean anything in the first place.

What I would take from this

Evidence that flatters you is usually evidence you built. The clean chart came from a mock I had shaped, without noticing, to produce it. The bigger score comes from aggregation that peeks at the future. In both cases the result looked like proof and was really a reflection of my own assumptions. Real data, and production, are the only tests that do not tell you what you want to hear.

A count of shared entities lies when the entity is low-cardinality. OS families, email domains, regions, anything a large share of your population has in common, will dominate the count and look like signal. Guard for it, or your reason codes will confidently accuse a Windows laptop.

Leakage-safe is a discipline, and some famous tricks break it. Strictly-before, current row excluded, no aggregation over data you would not have at decision time. The move that lifts your offline score by using the future is the move that fails silently in production.

Report the honest number and the operating result, not the vanity metric. A PR-AUC out of context invites a snap judgment. "Recovers 62% of fraud value at this review budget, here is the curve" invites a conversation with whoever owns the budget. The second one is the one that matters.

The code, the leakage guards, the investigation queries, and the full device-fingerprint write-up are in the repo: github.com/gbadedata/transaction-fraud-scoring.

The best result in the whole project was a wrong alert about a Windows laptop, because it was the moment the data stopped agreeing with me. That is usually the moment something true starts.

The hardest part of an autonomous AI agent is the unhappy path

Oluwagbade Odimayo — Fri, 03 Jul 2026 20:57:13 +0000

Most demos of AI agents show you the happy path: a clean question, a tidy answer, everyone claps. The interesting engineering is everywhere else. What does your agent do when the API it depends on is down? When the model would happily keep looping, and your credit card is attached to every step? When it has no data, but is perfectly capable of writing something that looks like data anyway?

I built an autonomous agent for a domain where those questions are not academic, and getting the unhappy path right turned out to be most of the work. Here is what I learned.

The project is github.com/gbadedata/bioagent.

What it does

BioAgent is an autonomous quality-control analyst for a genomics pipeline. You give it a sample ID and it does the rest on its own: it pulls concordance and reproducibility metrics from a live pipeline API through a set of tools, works out what the numbers mean against benchmark thresholds, builds a targeted PubMed query from the actual findings, searches the literature, and writes a structured, clinical-grade quality report. It streams the whole thing into a Streamlit chat as it reasons, and exposes a FastAPI endpoint a scheduler can call.

It is built with LangGraph and Claude. Why LangGraph, and not a plain "here is a list of tools" agent, is the whole point of this post.

Why a graph, and why bounded

A plain agent takes a question, maybe calls some tools, and answers. BioAgent has to make decisions in sequence: fetch data, then decide from what came back whether the literature is even worth searching; if the search is empty, broaden it and retry; if the pipeline is unreachable, stop and say so clearly. That is a state machine with cycles and conditional routing, which is exactly what LangGraph models.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#eef2f7','primaryBorderColor':'#1b2a4a','primaryTextColor':'#1b2a4a','lineColor':'#4c78a8','fontFamily':'Segoe UI, sans-serif'}}}%%
flowchart TD
    START([sample_id]) --> FETCH["fetch_data<br/>call 5 pipeline API tools"]
    FETCH -->|data collected| ANALYSE["analyse<br/>LLM builds a targeted PubMed query"]
    FETCH -->|"critical tools failed,<br/>retry budget remains"| FETCH
    FETCH -->|"critical tools failed,<br/>retries spent"| DEGRADE["graceful_degradation<br/>report what failed, invent nothing"]
    ANALYSE --> SEARCH["search_literature<br/>query PubMed, broaden and retry if empty"]
    SEARCH --> REPORT["synthesise_report<br/>LLM writes the QC report"]
    REPORT --> DONE([END])
    DEGRADE --> DONE

The property that matters most is that the graph is bounded. Every cycle has a hard retry limit; the agent physically cannot loop forever. When your agent calls paid APIs on every step, "cannot loop forever" is not a nice-to-have, it is a safety requirement.

And here is a single run, end to end:

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#eef2f7','actorBkg':'#eef2f7','actorBorder':'#1b2a4a','actorTextColor':'#1b2a4a','signalColor':'#4c78a8','signalTextColor':'#1b2a4a','noteBkgColor':'#f4f7fb','noteBorderColor':'#4c78a8'}}}%%
sequenceDiagram
    participant U as User / API
    participant G as LangGraph
    participant P as Pipeline API
    participant C as Claude
    participant L as PubMed
    U->>G: analyse(sample_id)
    G->>P: runs, concordance, reproducibility, alerts
    P-->>G: metrics (or structured errors)
    alt critical data still missing after a retry
        G-->>U: graceful-degradation report (no invented data)
    else data collected
        G->>C: build a PubMed query from the metric values
        C-->>G: query
        G->>L: search, broaden and retry if empty
        L-->>G: citations and abstracts
        G->>C: synthesise the QC report from data and abstracts
        C-->>G: structured report
        G-->>U: report, citations, and tool trace
    end

Lesson 1: bounding a loop is easy to get subtly wrong

Here is the routing after the data-fetch step:

def route_after_fetch(state):
    critical = {"get_concordance_summary", "get_pipeline_runs"}
    critical_failed = critical.intersection(set(state["failed_tools"]))

    if critical_failed and state["fetch_retries"] > MAX_FETCH_RETRIES:
        return "graceful_degradation"
    if critical_failed and state["fetch_retries"] <= MAX_FETCH_RETRIES:
        return "fetch_data"          # retry, bounded
    return "analyse"

The idea is simple: if the critical tools failed and there is retry budget left, try again; if the budget is spent, give up gracefully; otherwise carry on.

The subtlety is that a bound is only a bound if the counter actually moves. If the node doing the work forgets to increment the retry count, the router keeps seeing "budget remains" forever, and the graceful exit is never reached. The agent loops until the framework's recursion limit trips and throws, which is the exact opposite of failing safely. The fix is a single line in the fetch node:

return {
    ...
    "fetch_retries": state.get("fetch_retries", 0) + 1,   # the bound only works if this moves
}

This is the kind of bug that never appears while you are building, because you are always testing the happy path where the API is up. It only shows up when the dependency breaks. So the real fix is not the one-line increment, it is a test that runs the agent with the API forced down and asserts it degrades rather than loops:

def test_full_run_degrades_when_api_down(mock_pipeline_api_down):
    result = run_agent("HG001")
    assert result["status"] == "degraded"

Here is the full suite, including that degrade-not-loop test, running green in CI:

Test the unhappy path, or you have not tested the part that matters.

Lesson 2: an agent with no data must not write a report

The most dangerous failure for this kind of system is not a crash. It is a confident, clinical-looking report generated from nothing. So when the critical tools cannot be reached, the graph routes to a dedicated node that reports exactly which tools failed, says what could not be retrieved, tells you how to start the API, and stops. It never fills the gap with plausible numbers.

That is not just a prompt instruction, it is enforced by a test that asserts the degraded report contains no fabricated metrics:

def test_report_does_not_hallucinate_metrics():
    result = graceful_degradation(state_with_failed_tools)
    assert "0.99" not in result["report"]
    assert "F1"   not in result["report"]

A rule the model is asked to follow is a hope. A rule a test enforces is a guarantee.

Lesson 3: ground the model in what it actually retrieved

The agent cites PubMed papers in its report, which is a quiet invitation to invent relevance. Early on it fetched abstracts and then discarded them, passing only the PMIDs downstream, so the model was asked to explain how papers supported the findings without ever seeing what those papers said. That is exactly the kind of shortcut that produces confident nonsense.

The fix was to carry the retrieved abstract text through to the report step and tell the model to ground its literature section only in the abstracts provided, and to say so plainly if none were retrieved. Retrieval is only grounding if the retrieved text actually reaches the place that writes the words.

Takeaways

Bound every loop, and prove it. In an autonomous agent, an unbounded retry is a runaway bill. A bound only counts if the counter advances, so test that it does.
Test the unhappy path. The happy path is the part that was always going to work. Force the dependency down and assert the agent fails safely.
No data, no report. Make "do not invent" a tested guarantee, not a polite request in a prompt.
Grounding means the retrieved text reaches the writer. Fetching abstracts and then ignoring them is worse than not fetching at all.

The code, the full LangGraph state machine, the tests, and the architecture diagrams are all in the repo: github.com/gbadedata/bioagent. If you want the MCP-server-plus-tool-using-agent take on the same ideas, I wrote that up separately at github.com/gbadedata/mcp-research-agent.

I built a transformer from scratch to classify airline complaints. A TF-IDF baseline beat it.

Oluwagbade Odimayo — Thu, 02 Jul 2026 18:30:50 +0000

Customer feedback is one of the great unglamorous NLP problems. Millions of short, messy, opinionated messages, and someone has to turn them into "what is going wrong, and how often." I wanted to build the core of that pipeline properly, so I took about 14,600 real tweets about US airlines and set two tasks: classify each tweet's sentiment, and, for the complaints, classify the reason (late flight, lost luggage, customer service, and so on).

I gave myself one rule: build the transformer from scratch in PyTorch, and do not trust it until it has beaten a stupid-simple baseline.

It didn't beat the baseline. That turned out to be the most useful result in the project, and the rest of this post is about why.

The full code is here: github.com/gbadedata/airline-feedback-transformer.

The data and the two tasks

The dataset is the Twitter US Airline Sentiment set: real customer feedback about six airlines from February 2015, labelled for sentiment and, for the negative tweets, a complaint reason.

It is imbalanced the way genuine complaints are: 9,178 negative, 3,099 neutral, 2,363 positive. I kept that imbalance rather than resampling it away, because production feedback is imbalanced too, and hiding it just moves the problem.

Task 1, sentiment: three classes over all tweets.
Task 2, reason: the complaint reason over negative tweets that have an identifiable one. I dropped the "Can't Tell" bucket, since by definition it has no aspect to learn, which left nine classes over 7,988 tweets. This is the extraction-flavoured task.

Both use a stratified 70/15/15 split.

The simple baseline I had to beat

Before any deep learning, the thing to beat: TF-IDF features into a class-weighted logistic regression. This is a genuinely strong text classifier and it fits in a few lines.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

vec = TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True, max_features=40000)
clf = LogisticRegression(max_iter=2000, C=4.0, class_weight="balanced")
clf.fit(vec.fit_transform(train_texts), y_train)

Plus a majority-class floor, so I always know what "predicting nothing" scores.

The transformer, built from scratch

I could have called AutoModel.from_pretrained(...), but building the encoder by hand is a much better way to show the architecture is understood rather than imported: token and learned positional embeddings, multi-head self-attention with a correct padding mask, pre-norm residual blocks with a GELU feed-forward, masked mean pooling, and a small classification head.

The part that quietly matters most is the padding mask. Tweets are short, so batches are mostly padding, and if the model attends to pad tokens it learns from noise. Padded keys have to be removed before the softmax:

scores = (q @ k.transpose(-2, -1)) / math.sqrt(self.d_head)   # (B, heads, T, T)
scores = scores.masked_fill(pad_mask[:, None, None, :], NEG_INF)
attn = torch.softmax(scores, dim=-1)

and the same mask has to exclude padding from the pooled representation:

keep = (~pad_mask).float().unsqueeze(-1)                      # (B, T, 1)
pooled = (x * keep).sum(1) / keep.sum(1).clamp(min=1.0)       # (B, d_model)

I unit-tested both. One test adds extra padding to an input and asserts the prediction does not change, which is exactly the property a correct mask guarantees. The default model is deliberately compact (d_model 128, two layers, four heads, about 0.97M parameters) so it trains on a CPU in minutes.

Training it

A standard, explicit PyTorch loop: class-weighted cross-entropy for the imbalance, AdamW, linear warmup and decay, gradient clipping, and early stopping on validation macro-F1 with the best checkpoint restored.

loss_fn = nn.CrossEntropyLoss(weight=class_weights)          # counter the imbalance
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
# ... warmup+decay schedule, clip_grad_norm_, early stop on val macro-F1

One evaluation framework for everyone

Every model (majority, TF-IDF, transformer, and an optional zero-shot LLM) is scored by the same code: accuracy, macro-F1, weighted-F1, a full per-class precision/recall/F1 table, and a confusion matrix. Define the test set once, define the metrics once, run every model through them. That is the piece that makes the comparison trustworthy, and it is the piece that tells you where a model fails rather than just how often.

The results

Task	Model	Accuracy	Macro-F1
Sentiment	Majority	0.627	0.257
	TF-IDF + LogReg	0.790	0.735
	Transformer (scratch)	0.769	0.729
Reason	Majority	0.364	0.059
	TF-IDF + LogReg	0.648	0.501
	Transformer (scratch)	0.612	0.495

On both tasks the transformer lands within a single macro-F1 point of the baseline and does not pass it. It is not broken: it learns cleanly and then early-stops.

Where it wins and where it loses

The per-class view is where the evaluation framework earns its place. On the reason task the model is good at the frequent, distinctive complaints and poor at the rare ones.

Late Flight (0.72), Customer Service Issue (0.68), Lost Luggage (0.68) and Cancelled Flight (0.67) come out well. Damaged Luggage (0.17, with eleven test examples) and longlines (0.27) do not. The confusion matrix shows the errors are sensible: rare and adjacent complaints get pulled toward the big "Customer Service Issue" and "Late Flight" classes.

That is an actionable readout. It tells you which classes need more data or a different approach, which is far more useful than a single headline accuracy.

Why the baseline won, and why that matters

Transformers earn their advantage from pretraining on enormous corpora. Mine was trained from random initialisation on about 10,000 short tweets, which is nowhere near enough for the architecture to express its strengths. The bottleneck was never the model's capacity. It was data and pretraining.

So at this scale the correct engineering decision is the simpler, faster, more interpretable TF-IDF model, and the value of the whole exercise is being able to say that with numbers instead of assuming the neural model must be better. That is the judgment that separates "I can train a transformer" from "I know when a transformer is worth it."

What I would do in production

Nothing about the finding says transformers are the wrong tool. It says a from-scratch one is. The pipeline is built for the obvious next step: swap the scratch encoder for a pretrained one and fine-tune it. The pooling, the head, the evaluation framework and the training loop do not change, and that is where the neural approach starts to pull ahead. Parameter-efficient fine-tuning (LoRA) slots in at the same point.

Takeaways

Always build the baseline first. A class-weighted TF-IDF logistic regression is a high bar, and if your model cannot clear it, that is information, not failure.
Build the architecture by hand once. Implementing attention, masking and pooling yourself teaches you where the bodies are buried, and the padding mask is one of them.
Measure per class, not just overall. The headline number hides which complaint types you actually handle.
Report the result that does not flatter you. "The simple model won" is often the most valuable sentence in a project.

Code, tests and figures: github.com/gbadedata/airline-feedback-transformer. If you want the retrieval and RAG side of this kind of work, I wrote up a biomedical question-answering system separately at github.com/gbadedata/biomedical-rag-qa.

I Built a Biomedical RAG System, and a 40-Year-Old Algorithm Beat My Vector Database

Oluwagbade Odimayo — Thu, 02 Jul 2026 11:45:53 +0000

A hands-on walkthrough of a retrieval-augmented QA pipeline over PubMed abstracts, the evaluation that kept me grounded, and why BM25 out-retrieved a FAISS vector index.

Everyone reaches for a vector database the moment they hear "RAG". I did too. Then I measured it against a lexical baseline from the 1980s, and the baseline won on every metric.

This post walks through a small retrieval-augmented question-answering system I built over biomedical literature, the evaluation that produced that result, and the two lessons that mattered more than any model choice. The full code is on GitHub: gbadedata/biomedical-rag-qa.

What we are building

The task: given a clinical or biological question, retrieve the passages that bear on it and produce a grounded yes / no / maybe answer, with citations, rather than letting a language model answer from memory.

The pipeline is five small modules, each usable and testable on its own.

Ingest from public APIs, build a passage corpus, retrieve, generate a grounded answer, and evaluate every stage against a baseline.

I will focus on the two stages that produced the interesting results: retrieval and answering.

The data

I used PubMedQA (Jin et al., 2019): 1,000 expert-labelled biomedical questions, each paired with the abstract it was written from, already split into labelled sections, plus a yes / no / maybe decision.

To turn this into a retrieval benchmark, I treat each abstract section as a passage (3,358 in total, about 60 tokens each) and define a question's gold passages as the sections from its own abstract. A retriever's job is then to rank a question's gold passages against the whole pool. Clean, reproducible, and it lets several retrievers compete on identical ground.

Retrieval: three approaches, one interface

The key design choice is that the retriever is a swappable component, not a hard-wired call. Everything sits behind one interface, so I can benchmark a lexical method, a dense vector method, and a random floor without touching the rest of the pipeline.

class DenseRetriever:
    """TF-IDF -> truncated SVD (LSA) -> L2-normalise -> FAISS inner-product index."""
    name = "dense_lsa_faiss"

    def index(self, passages):
        self._ids = [p.passage_id for p in passages]
        mat = self._embed([p.text for p in passages], fit=True)  # (n, d) float32
        self._index = faiss.IndexFlatIP(mat.shape[1])
        self._index.add(mat)
        return self

    def search(self, query, k=10):
        q = self._embed([query], fit=False)
        scores, idx = self._index.search(q, k)
        return [(self._ids[i], float(s)) for i, s in zip(idx[0], scores[0]) if i != -1]

The dense retriever builds TF-IDF vectors, compresses them to 256 dimensions with truncated SVD (latent semantic analysis), normalises, and indexes them in FAISS. The lexical retriever is plain BM25. And there is a random retriever, because you always want to know the floor.

Here is what happened.

BM25 places a gold passage first for 94.3% of questions (MRR 0.959) and leads the dense LSA + FAISS index across the board. Random is the floor.

BM25 wins everywhere. Its MRR of 0.959 means the first relevant passage is almost always ranked first or second.

Why did the "simple" method win? Because PubMedQA questions are written from their source abstracts, so they share a lot of vocabulary with the passages that answer them, and lexical overlap is a very strong signal here. Compressing that into 256 LSA dimensions trades away precision the benchmark actually rewards.

There is a subtlety in recall@1 worth knowing. Each question has about 3.4 gold passages, so you can only ever retrieve one of them at rank 1. That caps mean recall@1 at 0.319. BM25 scores 0.300, which is 94% of the mathematical ceiling. Its top rank is almost always correct.

The lesson is not "BM25 is better than vectors". It is measure it. On a different corpus, with a biomedical transformer embedder instead of LSA, the result could flip. But you only know by benchmarking against a baseline, and the swap is a one-line change to DenseRetriever._embed.

Does retrieval actually help the answer?

Good retrieval is worth nothing if it does not improve the answer. So I ran a diagnostic: train a simple decision classifier on four feature sets and compare them against a majority-class baseline.

Question only (no retrieval)
Retrieved context (BM25 top-3)
Gold context (perfect retrieval, the ceiling)
The majority baseline itself

A linear reader lifts macro-F1 well above the baseline by learning the minority classes, but nothing beats the baseline's accuracy of 0.553.

This is where it got uncomfortable, and interesting. The classifier lifts macro-F1 from the baseline's 0.237 to about 0.41, but no condition beats the baseline's accuracy, and feeding it retrieved passages by naive concatenation actually hurt it. Break it down by class and the reason is clear:

Even with perfect context, a bag-of-words reader manages F1 0.67 on "yes" but only 0.20 on the ambiguous "maybe" class.

PubMedQA is deliberately built to require reasoning over evidence, and a bag-of-words linear model cannot reason. It handles the easy majority class and falls apart on the ambiguous one.

This is the second, bigger lesson: retrieval quality is necessary but not sufficient. The value of RAG shows up only with a reader capable of reasoning over the retrieved evidence. Which is exactly why the answer step in the pipeline is an LLM, not a classifier.

Grounded generation

The generation step is thin on purpose, and strict about grounding. The model must answer only from the numbered passages, cite them, and return machine-checkable JSON.

SYSTEM_PROMPT = (
    "You are a careful biomedical research assistant. Answer the question using ONLY "
    "the numbered passages provided. Decide yes, no, or maybe. Use 'maybe' when the "
    "passages are mixed or insufficient. Do not use outside knowledge. Reply as strict "
    'JSON with keys "decision", "justification", and "supporting_passages".'
)

That contract is what makes the output auditable: you can check that the cited passages exist, and later that they actually support the justification. It also stops the model quietly answering from training memory, which is the whole point of RAG in a domain where a wrong answer matters.

Engineering choices that paid off

A few decisions that are easy to skip and worth keeping:

Baselines everywhere. A random-retrieval floor and a majority-class floor. The random floor is how you catch a silent indexing bug; the majority floor is how you avoid celebrating a model that only learned the class balance.
One retriever interface. Swapping BM25 for a dense model, or LSA for transformer embeddings, is a local change. The index and search loop do not move.
Two API styles behind one schema. Ingestion pulls from Europe PMC (REST, cursor pagination) and ClinicalTrials.gov v2 (REST, token pagination), plus Open Targets (GraphQL), all normalised to one passage schema with retry and backoff.
Metrics that do not need the LLM. The reported numbers are retrieval metrics and a linear diagnostic, so anyone can reproduce them without an API key. The generation step is real and runnable, but I did not claim an accuracy number I could not reproduce cheaply.

Run it yourself

git clone https://github.com/gbadedata/biomedical-rag-qa
cd biomedical-rag-qa
pip install -r requirements.txt
python scripts/fetch_data.py
python -m biomedqa.cli eval --data data/ori_pqal.json     # reproduces the numbers above

The evaluation is deterministic, tests run in CI across Python 3.10 to 3.12, and the whole thing is MIT licensed.

Takeaways

Two things I will carry into the next RAG project. First, benchmark retrieval against a lexical baseline before you assume a vector database is buying you anything, because sometimes it is not. Second, retrieval and generation are separate problems: strong retrieval with a weak reader still fails, so measure them independently and put the reasoning where it belongs.

Code, tests and full results: github.com/gbadedata/biomedical-rag-qa. Questions and critique welcome.

If you want the classical-ML counterpart, I ran a similar teardown on 215,000 patient drug reviews, sentiment classification plus complaint mining, with the same emphasis on baselines and reporting the results that do not help: github.com/gbadedata/drug-review-nlp.

Credit risk is more than predicting default: building the full stack in Python (IFRS 9 ECL, scorecards, monitoring)

Oluwagbade Odimayo — Tue, 30 Jun 2026 16:32:43 +0000

Most credit-risk tutorials stop at "train a classifier to predict default." That is maybe a fifth of what a real credit-risk function does, and not the interesting fifth. So I built the rest, on 1.35 million real loans, as three connected projects:

an IFRS 9 expected credit loss engine (PD, LGD, EAD, staging, macro scenarios),
a Weight-of-Evidence scorecard plus an independent model validation, and
a portfolio monitoring and management-information pack.

Stack is deliberately boring and reproducible: pandas, scikit-learn, matplotlib. Data is the public Lending Club accepted-loans tape. This post walks the techniques and the decisions, with the code that matters and the gotchas that bit me.

The data and the splits

The tape is one file of ~2.26M loans. The trick is that it is really three populations, and you need different slices for different jobs:

CHARGED = {'Charged Off', 'Default', 'Does not meet the credit policy. Status:Charged Off'}
PAID    = {'Fully Paid', 'Does not meet the credit policy. Status:Fully Paid'}
ACTIVE  = {'Current', 'In Grace Period', 'Late (16-30 days)', 'Late (31-120 days)'}

df['completed'] = df['loan_status'].isin(CHARGED | PAID)   # for PD training
df['default']   = df['loan_status'].isin(CHARGED).astype(int)
df['active']    = df['loan_status'].isin(ACTIVE) & (df['out_prncp'] > 0)  # the live book

Two habits that run through everything:

Out-of-time validation, not random split. Train on older vintages, test on newer ones, because that is the only test that tells you how the model behaves on loans it has not seen.

dev = df[df['issue_year'] <= 2015]   # build here
oot = df[df['issue_year'] >= 2016]   # judge here

No leakage. I drop the platform's own grade and interest rate from every model, so it earns its signal from borrower attributes rather than copying someone else's pricing.

And one engineering note up front: the raw file is ~390MB gzipped, so read it in chunks with usecols to keep memory sane:

for ch in pd.read_csv(RAW, usecols=COLS, chunksize=300_000, low_memory=False):
    ...

Part 1: the IFRS 9 ECL engine

The whole thing reduces to one line, ECL = PD x LGD x EAD, discounted and summed, but each term is its own small project.

PD. A logistic model, validated out of time. The model gives a lifetime PD; for the 12-month figure that Stage 1 needs, convert it under a constant-hazard assumption over the remaining term:

pd_12m = 1 - (1 - pd_life) ** (np.minimum(12, rem_months) / rem_months)

LGD, measured not assumed. This is the bit most tutorials skip. For each charged-off loan, exposure at default is the principal still outstanding when it defaulted, and the recovery is the post-default cash, net of fees:

ead_at_default = (funded_amnt - total_rec_prncp).clip(lower=0)
lgd = (1 - recoveries / ead_at_default).clip(0, 1)
# result on this book: mean LGD = 0.91  (a 9% recovery rate)

That 0.91 is high because the loans are unsecured. On a secured (auto) book it would be much lower and more dispersed, and the whole allowance would shrink.

Staging and ECL. Stage 3 is impaired (31+ days past due), Stage 2 is significant-increase (arrears backstops plus a PD-based trigger), Stage 1 is the rest. Then:

ecl = np.where(stage == 1, ead * pd_12m  * lgd / (1 + eir),
      np.where(stage == 2, ead * pd_life * lgd / (1 + eir) ** (rem_yrs / 2),
                           ead * np.minimum(1.0, lgd) / (1 + eir) ** (rem_yrs / 2)))  # stage 3, PD=1

Result on the live book ($9.5bn EAD): ECL $1.25bn, coverage 13.1%, with coverage rising 6.6% -> 31% -> 77% across the three stages.

The gotcha worth knowing. The most consequential input is not a parameter, it is the Stage 2 trigger, because it swaps a 12-month provision for a lifetime one. I swept it:

for q in np.linspace(0.75, 0.95, 9):
    thr = act['pd_life'].quantile(q)
    stage = np.where(..., (act['pd_life'] > thr), ...)
    # total ECL ranges ~$1.10bn -> $1.37bn as you flag 5% -> 25% of the book as Stage 2

A third of a billion dollars hangs on one threshold. Worth knowing before you trust the headline number.

Part 2: a WoE scorecard, then break it

Scorecards in banking are not gradient-boosted black boxes; they are Weight-of-Evidence logistic models scaled to points, because they have to be explainable. WoE and Information Value:

woe = np.log((dist_good + eps) / (dist_bad + eps))
iv  = ((dist_good - dist_bad) * woe).sum()   # select features by IV, e.g. >= 0.02

Fit a logistic on the WoE values, then scale to points (the classic PDO formulation, 20 points to double the odds, anchored at 600 for 50:1):

factor = PDO / np.log(2)
offset = BASE - factor * np.log(BASE_ODDS)
points = -(coef * woe + intercept / n) * factor + offset / n   # per characteristic

Now the part that matters more than the build: validation. Four tests on the out-of-time sample.

1. Discrimination holds: Gini 0.356 out of time (0.385 in development), KS 0.256.

2. Calibration is where it nearly slipped. Discrimination tells you the ranking is right; it says nothing about whether the predicted probability is right. Check the level:

pred_over_observed = oot['pd'].mean() / oot['target'].mean()   # 0.77

0.77 means the model under-predicts default by ~23% on recent vintages, on every score band. Fine for ranking, not safe for pricing or ECL until recalibrated.

3. The trap inside the trap: PSI vs calibration. Population Stability Index checks whether the applicant mix shifted:

psi = (((dev_pct - oot_pct) * np.log((dev_pct + eps) / (oot_pct + eps)))).sum()   # 0.013, stable

PSI was a tiny 0.013. A monitor watching only PSI would flash green while the model quietly went biased, because a stable population does not mean an accurate model. Different questions; check both.

4. Effective challenge. Benchmark against a HistGradientBoostingClassifier on raw features: Gini 0.401 vs the scorecard's 0.356. A small lift, not enough to justify losing the transparency, so it becomes a watch item rather than a rebuild.

Verdict: approved with conditions, written up as a RAG-rated validation report. The deliverable of a second line is not a model, it is a defensible opinion.

Part 3: monitoring the live book

A point-in-time book can look healthy while deteriorating, because delinquency lags. The leading view is the vintage curve: group loans by origination year and track cumulative default by months on book.

# default timing approximated from last payment date
mob_default = last_pymnt_month - issue_month          # months on book at default
cum_default = [(coh_mob <= m).sum() / n * 100 for m in range(25)]   # per cohort

This surfaced the signal the delinquency rate hid: the 12-month-on-book default rate rose from 4.7% (2013) to 6.8% (2016), with the recent cohorts sitting above the older ones at every age.

Then scorecard drift over time (PSI by vintage vs a baseline, which climbed to 0.13), and concentration via a Herfindahl index:

shares = exposure.groupby('addr_state').sum() / exposure.sum()
hhi = (shares ** 2).sum()   # 0.051 by state (diversified); 58% in one product (watch)

Everything rolls into a RAG early-warning dashboard. The output is intentionally mixed: current losses green, vintage trend red, model drift and product concentration amber. An all-green dashboard on a quietly worsening book is the failure mode.

Gotchas, collected

Read the big gz in chunks; do not load 390MB into a single frame.
Exclude the platform's grade/rate or your model just relearns someone else's pricing.
Recent vintages are right-censored in the vintage curves; show them, but read them as partial.
Goodness-of-fit stats are useless at 500k rows (they reject on noise); use the predicted/observed ratio and a calibration plot instead.
Discrimination is not calibration, and PSI is neither. Three different questions, three different checks.
LGD of 0.91 is a feature of unsecured lending, not a bug; a secured book changes the whole picture.

The repos

Each is standalone, reproducible (pip install -r requirements.txt && python analysis.py), and ships an executed notebook plus a written report:

IFRS 9 ECL engine: github.com/gbadedata/ifrs9-ecl-engine
Scorecard + independent validation: github.com/gbadedata/pd-scorecard-validation
Portfolio monitoring + MI pack: github.com/gbadedata/credit-risk-monitoring

Built on public US consumer data; the methods carry directly to a secured book such as auto finance, where the parameters (recovery above all) would differ. If you build something similar, I would be glad to compare notes in the comments.

Building an Affordability-First Credit Stack: Three ML Projects on Real Lending Data

Oluwagbade Odimayo — Mon, 29 Jun 2026 22:19:01 +0000

The engineering behind three fintech machine-learning projects: behavioural default risk, affordability-based risk, and Open Banking transaction categorisation, with the decisions that actually matter."

Most "predict loan default" tutorials make the same three mistakes: they report accuracy on an imbalanced target, they leak future information into the features, and they stop at a probability instead of a decision. This write-up is about avoiding all three, across three projects on real lending data that together build toward an affordability-first view of credit risk.

A companion narrative piece covers why this matters. This one is about how: the data wrangling, the feature engineering, the experiment design, and the production logic. All the code is real, lifted from the repositories linked at the end.

The throughline

Three projects, three datasets, one argument:

Behavioural default risk on 30,000 real credit-card customers: the traditional credit-history approach, done with the right metrics and a cost-based decision.
Affordability-based risk on 1.35 million real Lending Club loans: a controlled test of whether affordability out-predicts a credit score.
Transaction categorisation on 259,000 real bank transactions: the Open Banking data layer that turns a raw bank feed into the income and spending signals affordability needs.

Lesson 1: under imbalance, accuracy is a trap

Roughly one borrower in five defaults in these datasets. A model that predicts "everyone repays" scores about 80% accuracy and is worthless. So across all three projects the metrics are precision-recall (average precision), the KS statistic, and recall at a chosen operating point, never raw accuracy.

The precision-recall curve is the meaningful picture because the no-skill baseline is not 0.5, it is the prevalence:

from sklearn.metrics import average_precision_score, roc_auc_score
# PR-AUC floor is the positive rate (~0.20), not 0.5
print(f"PR-AUC {average_precision_score(y_test, probs):.3f}  "
      f"(no-skill = {y_test.mean():.3f})")

For the credit-card model this is the difference between a believable ROC-AUC of 0.78 / PR-AUC 0.56 and a meaningless "97% accurate" headline.

Lesson 2: a probability is not a decision

A risk score becomes useful only when you decide where to cut. That cut is a business choice, because the two errors cost different amounts: a missed default loses the loan, while a wrongly declined good customer only loses the margin. So instead of defaulting to 0.5, sweep the threshold to minimise expected cost.

def optimal_threshold(probs, fn_to_fp_ratio):
    thresholds = np.linspace(0.01, 0.99, 99)
    costs = [
        fn_to_fp_ratio * ((probs < t) & (y_test == 1)).sum()  # missed defaults
                       + ((probs >= t) & (y_test == 0)).sum()  # good customers declined
        for t in thresholds
    ]
    return thresholds[int(np.argmin(costs))]

The threshold moves with the cost ratio, and so does the business outcome: at a 10:1 ratio the credit-card model catches 92% of defaulters; at 2:1 it approves far more and catches half. There is no single correct cutoff without a cost view, and the valuable artefact is this curve, not the raw probability.

Lesson 3: leakage is how credit models cheat

This is the big one, and it is the whole reason the affordability project's numbers are trustworthy. The Lending Club file has 151 columns, and many of them are recorded after the loan runs: total payments received, recoveries, the latest FICO pull. Train on those and you get a spectacular AUC that collapses in production, because at decision time they do not exist.

So the rule is strict: keep only what a lender knows at origination. Every post-loan field is dropped, and so are Lending Club's own grade and int_rate, because those already encode its internal risk model and would short-circuit the experiment.

BAD  = {"Charged Off", "Default", "Does not meet the credit policy. Status:Charged Off"}
GOOD = {"Fully Paid", "Does not meet the credit policy. Status:Fully Paid"}

# the raw file is 1.6 GB, so stream it in chunks and keep only finished loans
parts = []
for chunk in pd.read_csv(path, usecols=ORIGINATION_COLS,
                         compression="gzip", chunksize=300_000):
    done = chunk[chunk.loan_status.isin(BAD | GOOD)].copy()
    done["default"] = done.loan_status.isin(BAD).astype("int8")
    parts.append(engineer(done))
df = pd.concat(parts, ignore_index=True)   # ~1.35M completed loans

Two things worth noting: usecols plus chunksize keeps a 1.6 GB file inside a few hundred MB of RAM, and filtering inside the loop means you never materialise the rows you do not need.

Lesson 4: engineer affordability properly

Affordability is a ratio of obligations to income, so the features have to express that, and they have to be correct for joint applications (where two people share the liability and one person's income understates capacity).

joint = df.application_type.eq("Joint App")
df["income"]  = np.where(joint & df.annual_inc_joint.notna(),
                         df.annual_inc_joint, df.annual_inc)
df["dti_eff"] = np.where(joint & df.dti_joint.notna(),
                         df.dti_joint, df.dti)

df["payment_to_income"] = df.installment * 12 / df["income"]   # annual burden
df["loan_to_income"]    = df.loan_amnt / df["income"]

These three (payment-to-income, debt-to-income, loan-to-income) each produce a clean monotonic default gradient on their own, before any model touches them.

Lesson 5: design the experiment so the result means something

The headline claim, that affordability out-predicts a credit score, is only credible if the comparison is controlled. So I held the algorithm fixed and changed only the feature set: credit-and-bureau features, affordability features, then both. Any difference in performance is then attributable to the features, not to model tuning.

AFFORD = ["income", "dti_eff", "payment_to_income", "loan_to_income",
          "installment", "loan_amnt", "term_months", "emp_length_num"]
CREDIT = ["fico", "delinq_2yrs", "inq_last_6mths", "revol_util",
          "open_acc", "pub_rec", "total_acc", "credit_history_yrs"]

def evaluate(features):
    model = XGBClassifier(n_estimators=300, max_depth=5, learning_rate=0.05,
                          scale_pos_weight=spw, tree_method="hist", n_jobs=4)
    model.fit(X_train[features], y_train)
    p = model.predict_proba(X_test[features])[:, 1]
    return roc_auc_score(y_test, p), average_precision_score(y_test, p)

evaluate(CREDIT)            # ROC-AUC 0.617
evaluate(AFFORD)           # ROC-AUC 0.699  <- affordability alone wins
evaluate(AFFORD + CREDIT)  # ROC-AUC 0.706

tree_method="hist" is what makes XGBoost comfortable on 1.35 million rows, and scale_pos_weight set to the negative/positive ratio handles the imbalance without resampling. The result holds up: affordability beats the bureau record, and the two combined beat either alone.

Lesson 6: cleaning real transaction text

The categorisation project is a different kind of engineering. Real bank descriptions are hostile: Earnin PAYMENT Donatas Danyal, transfers buried in authorisation codes, dates and reference numbers everywhere. None of that noise carries category signal, so it gets stripped before vectorising.

import re

def clean(s):
    s = re.sub(r"\d+", " ", str(s).lower())   # auth codes, dates, amounts
    s = re.sub(r"[^a-z\s]", " ", s)           # punctuation
    s = re.sub(r"\b\w{1,2}\b", " ", s)        # 1-2 char fragments
    return re.sub(r"\s+", " ", s).strip()
# 'Earnin  PAYMENT  Donatas Danyal' -> 'earnin payment donatas danyal'

Then TF-IDF over unigrams and bigrams, with transaction amount stitched on as a numeric feature, because payroll and loan amounts are large and coffees are small:

from scipy.sparse import hstack, csr_matrix
tfidf = TfidfVectorizer(ngram_range=(1, 2), min_df=3,
                        max_features=30_000, sublinear_tf=True)
X_text = tfidf.fit_transform(df.clean_desc)
X = hstack([X_text, csr_matrix(scaled_log_amount)])

A class-weighted linear SVM on this reaches 0.80 macro-F1 across 31 categories (baseline 0.01), and crucially it is fully inspectable: reading the top-weighted terms per class shows it learned that mcdonald means restaurants and uber/lyft in incoming payments mean gig income. Macro-F1, not accuracy, again, because the categories are heavily imbalanced.

The confusions are reassuring rather than alarming: the various transfer types blur into each other because their text genuinely overlaps, which is label ambiguity, not model failure.

Lesson 7: turn confidence into an operating policy

A model that labels everything is a research artefact. A model that knows when to defer is a system. For the linear SVM, the gap between the top two class scores is a usable confidence signal:

margins = clf.decision_function(X_test)          # (n_samples, n_classes)
top2 = np.partition(margins, -2, axis=1)
confidence = top2[:, -1] - top2[:, -2]           # best minus runner-up
order = np.argsort(-confidence)                  # most confident first
correct = (preds == y_test)[order]
# cumulative accuracy as coverage grows -> the auto-classify / review trade-off

Auto-classifying the most-confident 80% holds 98% accuracy, sending only a fifth to human review. That single curve is what turns the categoriser into something a lending operation could actually deploy.

Lesson 8: explainability is non-negotiable

Regulated lending has to justify decisions, so every model here is explained with SHAP, at both the portfolio level and the individual decision. The single-applicant view is the one a compliance team cares about:

import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_sample)
# waterfall for one applicant -> exactly what an adverse-action notice needs

On the credit-card model, this produces a legible story for a single 97%-risk applicant: behind in every month, two months behind most recently, repaying almost nothing. Not a black box, a defensible decision.

Putting it together

The engineering themes repeat across all three projects, and they are the transferable part:

Judge models on the metric the problem demands (PR-AUC and KS under imbalance, macro-F1 across many classes), never on accuracy.
Treat the threshold as a cost decision, not a default.
Be ruthless about leakage; it is the single biggest reason credit models look great offline and fail in production.
Design comparisons so the result is attributable to the thing you are testing.
Engineer for the data you actually have, whether that is a 1.6 GB file that needs streaming or transaction text that needs aggressive cleaning.
Ship the human-in-the-loop logic, because confidence-based routing is what separates a model from a system.

Each project stands alone, but the line they trace, from credit history to affordability to the transaction data that makes affordability computable, is the actual shape of modern credit decisioning.

Full code, data notes and reproducible notebooks: credit-default risk, affordability-based risk, transaction categorisation.

From Global Averages to Producer-Level Variation: 3 Python Data Projects on Food's Climate Impact

Oluwagbade Odimayo — Mon, 29 Jun 2026 18:10:35 +0000

I spent a few weeks building three connected data-science projects on the greenhouse-gas footprint of what the world eats. They're deliberately a sequence - each one attacks an assumption the previous one had to make - and together they turned into a tidy case study in a few things I care about as a developer: benchmarking against baselines, respecting system boundaries, filtering data artefacts, and not letting a clean story beat an honest one.

This post is the technical tour: the data, the decisions, the code, and the gotchas. Full notebooks and data are in the repos linked at the end.

Stack: Python · pandas · NumPy · scikit-learn · Matplotlib/Seaborn · Jupyter.

The data (and a word on provenance)

Three public sources:

OECD–FAO Agricultural Outlook - meat consumption by country/year/type.
Our World in Data (CC BY 4.0) - national emissions; and the Poore & Nemecek (2018, Science) life-cycle emission factors.
FAOSTAT (CC BY 4.0) - country-level emission intensities of livestock products.

The repos ship the data with a Data licence & attribution note distinguishing the MIT-licensed code from the data, which keeps its own terms. If you publish someone's dataset, do this.

Project 1 - A footprint metric, diet clustering, and the model that lost

Goal: turn demand into estimated emissions, find diet archetypes, and forecast where consumption is heading.

The metric itself is trivial - and that's the point; the value is in operationalising it cleanly:

# Mean life-cycle GHG, kg CO2e per kg of product (Poore & Nemecek via OWID)
EF = {"beef": 99.5, "sheep": 39.7, "pig": 12.3, "poultry": 9.9}
df["footprint_kt_co2e"] = df["volume_kt"] * df["type"].map(EF)

That alone surfaces the headline: in 2020 beef was ~21% of meat volume but ~68% of the footprint. A minority of volume drives the majority of emissions.

Clustering countries by their meat mix (not absolute volume) separates two levers people conflate - the carbon intensity of the mix vs the quantity eaten:

shares = volumes.div(volumes.sum(axis=1), axis=0)        # per-country type shares
X = StandardScaler().fit_transform(shares)
labels = KMeans(n_clusters=k, n_init=10, random_state=42).fit_predict(X)
# archetypes range ~27 (poultry-led) to ~59 (beef/sheep-led) kg CO2e per kg of meat

The most useful thing I did: benchmark the forecast against a dumb baseline

It's easy to fit a trend, plot it, and declare victory. So I held out 2015–2019 and pitted a linear trend against a naive "next year = this year" baseline:

train = s[s.year <= 2014]
test  = s[s.year.between(2015, 2019)]

naive_pred = train.value.iloc[-1]                        # last observed value
naive_mae  = (test.value - naive_pred).abs().mean()

coef       = np.polyfit(train.year, train.value, 1)      # linear trend
trend_mae  = np.abs(test.value.values - np.polyval(coef, test.year)).mean()

print(f"naive MAE={naive_mae:.2f}  trend MAE={trend_mae:.2f}")
# total meat -> naive 0.17 vs trend 4.10   |   poultry -> naive 0.52 vs trend 1.24

The naive baseline won by a mile. The series had flattened after ~2014, so a trend fit on the earlier rise overshot. Lesson #1: a model that can't beat the naive baseline shouldn't be used - and you report that, you don't bury it.

Project 2 - Opening up the single carbon number

Project 1 used one global factor per food. Project 2 asks what that hides, using the Poore & Nemecek per-product dataset (43 foods, GHG split into seven supply-chain stages plus four other impacts).

First, a free data-integrity check - the stages should sum to the reported total:

stages = ["land_use_change","feed","farm","processing","transport","packaging","retail"]
assert np.allclose(df[stages].sum(axis=1), df["total_ghg"], atol=1e-6)   # passes to ~0 error

The "food-miles" myth, in three lines

animal = df[df.category == "animal"]
transport_pkg = animal[["transport","packaging"]].sum().sum()
print(transport_pkg / animal["total_ghg"].sum())          # ~0.047

Transport + packaging is ~4.7% of animal-product emissions; for beef, transport alone is ~0.5%. ~90% is farm + feed + land-use change. "Buy local" is a weak lever next to "change what you eat."

Is carbon a good proxy for everything? Not water.

df[["ghg","land","freshwater","eutrophication"]].corr()
# ghg<->land 0.83 | ghg<->eutrophication 0.76 | ghg<->freshwater 0.33

Carbon tracks land and nutrient pollution well, freshwater poorly (some plants - nuts, rice - are the thirstiest foods). A carbon-only metric can hide a water trade-off. PCA backs this up: PC1 explains ~66% of variance (an overall-impact axis), PC2 ~21% (a distinct water axis).

Z = StandardScaler().fit_transform(df[impact_cols])
print(PCA(n_components=2).fit(Z).explained_variance_ratio_)   # [0.66, 0.21]

Re-basing per 100 g of protein (the fair comparison, restricted to genuine protein sources) keeps the animal/plant gap enormous - beef ~50–100× pulses or nuts, with eggs the most efficient animal protein. Lesson #2: measure more than one dimension, or you'll move harm instead of removing it.

Project 3 - The same product, ~70× variation (and the messy data work behind it)

Both earlier projects used global means. Project 3 quantifies what they flattened, using FAOSTAT emission intensities (kg CO₂e/kg) for livestock products across 250 areas, 1961–2023. This is where most of the real engineering lived.

Gotcha 1 - encoding. Read it as latin-1 and Türkiye becomes TÃ¼rkiye. The file is UTF-8:

df = pd.read_csv(SRC, encoding="utf-8")   # not latin-1

Gotcha 2 - aggregates masquerading as countries. FAOSTAT mixes regional aggregates (World, Africa, income groups) into the same Area column. They use Area Code ≥ 5000:

AGG = ["World","Africa","Americas","Asia","Europe","Oceania","income",
       "European Union","Least Developed","Sub-Saharan", ...]
is_country = (df["Area Code"] < 5000) & \
             ~df["Area"].str.contains("|".join(AGG), case=False, na=False)

Gotcha 3 - data artifacts inflating the headline. My first pass gave a beef max/min ratio of 589× - driven by places that barely raise cattle (Hong Kong, Lebanon) reporting unreliable near-zero intensities. The fix is a per-product production floor, which also dropped a log(0) that was crashing the clustering:

MINPROD = {"Beef":50_000, "Cow milk":100_000, "Chicken":50_000,
           "Eggs":30_000, "Pork":50_000, "Sheep meat":10_000}   # tonnes/yr

def substantial(product):
    s = intensity[product]
    keep = (production[product].reindex(s.index).fillna(0) >= MINPROD[product]) & (s > 0)
    return s[keep].dropna()

With genuine producers only, the spread is real, not noise: ~70×, from ~3.9 kg CO₂e/kg (Israel) to ~270 (Niger).

Gotcha 4 - log before you cluster. Intensities are strongly right-skewed, so Euclidean k-means on raw values is dominated by the tail. Log-transform first:

X = intensity[feats].replace(0, np.nan).dropna()
Z = StandardScaler().fit_transform(np.log(X))             # log: intensities are right-skewed
labels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(Z)
# three efficiency tiers: median beef ~16 / 50 / 79 kg CO2e/kg

For the time trend, FAOSTAT publishes a World aggregate, so you don't have to re-derive it:

world = df[(df.Area == "World") & (df.Element == "Emissions intensity")]
# world beef -32% since 1961, cow milk -52%, chicken -41% as systems got more productive

The trap that ties it together: system boundaries

Here's the one that'll bite you if you're not careful. Across these projects, beef shows up as ~99.5, ~60, and ~30 kg CO₂e/kg. Same animal, three numbers:

~99.5 - OWID's headline life-cycle figure (Project 1).
~60 - the same Poore & Nemecek study, summed across its stages (Project 2).
~30 - FAOSTAT's farm-gate intensity (Project 3).

None of them are "wrong." They're measured to different system boundaries (full LCA vs production-only) and processed differently. Lesson #3: never compare emission factors across boundaries as if they're the same measurement. Lean on relative rankings (beef ≫ poultry) - those are stable across all three. I called this out explicitly rather than quietly picking whichever number suited the slide.

Engineering takeaways

Benchmark against a naive baseline. If your model can't beat "same as last value," it adds nothing - and saying so is the rigorous move.
Global means hide variation. For anything about intervention, the distribution is the unit of analysis, not the average (~70× for beef).
Mind system boundaries. 99 vs 60 vs 30 for the same animal. Compare like with like; trust relative rankings over absolute numbers.
Filter artifacts before quoting extremes. A production floor turned a noisy 589× into a real 70×.
Log-transform right-skewed features before distance-based methods (clustering, PCA).
Separate aggregates from units in mixed-grain datasets (FAOSTAT Area Code ≥ 5000).
Encoding is not optional (utf-8, or your country names rot).
Reproducibility hygiene: pinned requirements.txt, executed notebooks committed, data licensed and attributed.

Repos

meat-carbon-footprint - demand, the footprint metric, diet clustering, the baseline-beats-model forecast.
food-environmental-footprint - supply-chain stage decomposition, multi-impact correlations, PCA, per-protein.
livestock-emission-intensity - FAOSTAT cross-country variation, efficiency tiers, the system-boundary write-up.

Each has the full Jupyter notebook (executed), a runnable analysis.py, pinned requirements, and a detailed write-up.

If you take one thing from this: the interesting question is almost never the average. It's the spread, the boundary, and whether your model actually beats doing nothing. Happy to talk methods in the comments.

When SuSiE Says '95% Confident', Is It?

Oluwagbade Odimayo — Sun, 21 Jun 2026 23:12:05 +0000

Benchmarking the Honesty of Fine-Mapping Credible Sets

Fine-mapping has a promise built into its output, and almost nobody checks whether the promise is kept.

When you run SuSiE on a GWAS locus, it hands you a credible set: a small group of variants that, at a stated confidence level like 95%, should contain the true causal variant. That 95% is a claim about reality. Among all the loci where SuSiE reports a 95% credible set, the true causal variant should be inside the set 95% of the time.

Is it? This post is about how to measure that, what I found when I did, and why the answer is more interesting than a single coverage number.

Why you cannot measure this on real data

Here is the catch that shapes everything. To check whether a credible set contains the causal variant, you need to know the causal variant. On real GWAS data, you do not. That is the entire reason fine-mapping exists.

So calibration is measured by simulation, and this is not a shortcut, it is the only valid method. You plant a known causal variant in a simulated locus with realistic linkage disequilibrium, generate GWAS summary statistics consistent with that truth, run SuSiE, and check whether its credible sets behave as advertised. This is exactly how SuSiE, FiniMOM, SuSiEx, and the recent SuSiE 2.0 were all validated. Known ground truth is the whole point.

(To keep the simulation anchored to reality, the project also runs SuSiE on the SORT1 / 1p13 cholesterol locus, one of the rare real loci where the causal variant, rs12740374, is functionally validated. SuSiE recovers it. But the calibration numbers themselves come from simulation, as they must.)

The simulation

Each simulated locus needs three things SuSiE-RSS consumes: z-scores, an LD matrix, and a sample size.

simulate_locus <- function(p = 600, n = 20000, n_causal = 1,
                           pve = 0.002, block_w = 0.9, seed = 1) {
  set.seed(seed)
  R <- make_ld_matrix(p, block_w = block_w, neighbourhood = 6)
  causal <- sort(sample.int(p, n_causal))
  ncp <- sqrt(n * pve)
  b <- numeric(p); b[causal] <- ncp / sqrt(n_causal)
  mu <- as.vector(R %*% b)              # LD spreads the signal to proxies
  z <- as.vector(MASS::mvrnorm(1, mu = mu, Sigma = R))
  list(z = z, R = R, causal = causal)
}

The causal variant gets a true effect; LD spreads marginal signal to its neighbours (which is what makes fine-mapping hard); and the z-scores are drawn consistent with that LD structure. The effect size pve is tuned to the realistic marginal regime, mean absolute z around 6 at the causal variant, just above genome-wide significance. Make the signal too strong and fine-mapping becomes trivial; this is where it is actually interesting.

The metric, and validating the metric

Coverage is simple to state: across many loci, the fraction whose reported credible set contains the true causal variant. For a calibrated method at the 95% level, that should be about 0.95.

But a benchmark whose own metric is wrong is worse than no benchmark. So before trusting the metric on SuSiE, I tested it against a mock fine-mapper with known coverage: if you feed it credible sets that contain the causal variant exactly 95% of the time, the metric must report 0.95; feed it 80%, it must report 0.80.

mock <- function(n, true_cov) as.logical(runif(n) < true_cov)
empirical_coverage(mock(5000, 0.95))   # ~0.95
empirical_coverage(mock(5000, 0.80))   # ~0.80, correctly flagged

Only once the metric provably recovers known coverage is it allowed to judge SuSiE. (This logic was prototyped and unit-tested in Python first, then ported to R, so the arithmetic was known-correct independently of any fine-mapping run.)

The bug that inflated every credible set

The first real run produced credible sets of 100-plus variants. That is nonsense, no useful fine-mapping returns a 100-variant "credible" set. The cause was a single missing argument.

SuSiE's credible-set construction includes a purity filter: it prunes sets down to variants that are genuinely correlated with one another, discarding uncorrelated noise. That filter only runs if you pass the LD matrix to susie_get_cs:

# wrong: no purity filter, sets fill with uncorrelated noise
cs <- susie_get_cs(fit, coverage = 0.95)

# right: Xcorr = R activates the purity filter
cs <- susie_get_cs(fit, Xcorr = R, coverage = 0.95)

With the fix, sets collapsed from 100-plus variants to typically one. I found this by running single loci through SuSiE and inspecting the actual set sizes, rather than trusting an assumption about what they should be. The lesson is one the benchmark itself preaches: distrust a number that looks wrong, and check it against ground truth.

The result

Across thousands of loci, six difficulty conditions:

Condition	Coverage	Abstention
Baseline (1 causal)	100.0%	6%
Strong LD	100.0%	5%
Two causal	99.8%	41%
Three causal	99.7%	65%
Weak effect	99.4%	64%
Very weak effect	98.7%	85%

Coverage holds at or above the promised 95% everywhere, slightly conservative, which is the safe direction. So far, unremarkable: SuSiE is well-calibrated, as you would hope.

The interesting column is the second one.

The finding is in the abstention

Look at what happens as the loci get harder. SuSiE does not start returning wrong credible sets. It starts returning no credible set at all. Abstention climbs from 6% on clean single-causal loci, to 41% with two causal variants, to 85% in the lowest-power regime.

This is the mechanism behind the calibration. SuSiE keeps its sets sharp, usually a single variant, and protects its 95% promise by declining the loci where it cannot meet that bar. The high coverage is not achieved by hedging with big sets. It is achieved by abstaining on the hard cases instead of guessing.

That distinction matters enormously for how you read the output. A naive reading of "SuSiE returned no credible set" is failure. The correct reading is honesty: the method is telling you it cannot confidently localize the causal variant here, which is exactly what you want it to say when that is true. A fine-mapper that always returned a confident set, including on the loci where it has no business being confident, would be far more dangerous.

Why I think this generalizes

I have now built three benchmarks that ask the same question of very different systems: is the model's stated confidence honest? An LLM interpreting clinical variants abstained to "uncertain" exactly where the evidence ran out. A variant caller's QUAL scores could be checked for whether the stated confidence matched empirical precision. And here, SuSiE abstains rather than mislead as the genetics gets harder.

Three systems, one property worth measuring: not just whether the model is accurate, but whether it knows and admits the limits of what it knows. In all three, the well-built method's answer to "are you sure?" turns out to be the most informative thing it produces.

Run it

The full framework is R, built on susieR, with the simulation harness, the calibration metrics, the validated mock test, the figures, and the SORT1 real-locus template:

github.com/gbadedata/finemap-calibration-benchmark

Rscript setup.R                 # susieR, MASS, jsonlite
Rscript tests/test_calibration.R  # metric validation, no susieR needed
Rscript R/run_benchmark.R       # the full benchmark
Rscript R/make_figures.R        # the figures

If you fine-map, try measuring your own pipeline's credible-set coverage by simulation. The coverage number is reassuring. The abstention behaviour is where you learn what your method actually does when the data gets hard.

References

Verified against the primary source.

Wang G, Sarkar A, Carbonetto P, Stephens M (2020). A simple new approach to variable selection in regression, with application to genetic fine mapping. JRSS-B 82(5):1273-1300. doi:10.1111/rssb.12388
Zou Y, Carbonetto P, Wang G, Stephens M (2022). Fine-mapping from summary data with the "Sum of Single Effects" model. PLOS Genetics 18(7):e1010299. doi:10.1371/journal.pgen.1010299
Musunuru K, et al. (2010). From noncoding variant to phenotype via SORT1 at the 1p13 cholesterol locus. Nature 466(7307):714-719. doi:10.1038/nature09266

Your Variant Caller Tells You How Confident It Is. Have You Ever Checked If It's Telling the Truth?

Oluwagbade Odimayo — Sun, 21 Jun 2026 17:47:45 +0000

Every variant caller you have ever used attaches a number to each call: QUAL. It is a confidence score, a claim about how likely the call is to be correct. You filter on it constantly. QUAL >= 30, QUAL >= 20, whatever your pipeline settled on years ago.

Here is a question almost nobody asks: is that number honest?

When the caller stamps a variant at QUAL 30, it is claiming the call is 99.9% likely to be real. Among all the calls it stamps at that confidence, are 99.9% of them actually correct? Or is the caller systematically overstating how sure it is, and if so, where?

This is the question of calibration, and it is borrowed from machine-learning evaluation, where it is standard practice and where, for variant callers, it is almost completely absent. This post is about why it matters, how to measure it, and a small open-source framework that does.

What QUAL actually claims

QUAL is phred-scaled. The definition is precise:

QUAL = -10 * log10(P(call is wrong))

Invert it and you get the caller's stated probability that the call is correct:

def qual_to_confidence(qual: float) -> float:
    if qual <= 0:
        return 0.0
    p_wrong = 10.0 ** (-qual / 10.0)
    return 1.0 - p_wrong

So QUAL 10 claims 90% confidence, QUAL 20 claims 99%, QUAL 30 claims 99.9%. These are not vague quality hints. They are probability statements, and probability statements can be checked against reality.

Calibration: comparing the claim to the truth

To check them you need ground truth. For human variant calling that means the Genome in a Bottle (GIAB) benchmark set, the community-standard truth set for samples like HG001/NA12878. Run your caller, compare each call against GIAB: true positive if it is in the truth set, false positive if it is not.

Now group the calls by their stated confidence and, in each group, measure the empirical precision, the fraction that are actually true positives. A well-calibrated caller has, in every bin, empirical precision close to stated confidence. Plot one against the other and an honest caller's points sit on the diagonal.

The scalar summary of how far off you are is Expected Calibration Error (ECE): the support-weighted mean absolute gap between stated confidence and empirical precision across the bins.

ece = 0.0
for b in bins:
    gap = abs(b.mean_confidence - b.empirical_precision)
    ece += (b.n / total) * gap

ECE of 0 is perfect honesty. A large ECE with stated confidence consistently above empirical precision means the caller is overconfident: its QUAL says more than the calls deliver.

Why this is not academic

If QUAL is inflated, every threshold built on it is wrong, and wrong in a way that hides. Filter at "QUAL >= 30, surely safe" and, if the caller is overconfident, you are quietly keeping false positives you think you excluded. Worse, the inflation is rarely uniform. It concentrates in the hard parts of the genome, low-complexity regions, segmental duplications, low-mappability stretches, which is exactly where clinically important variants sometimes live and exactly where you can least afford a false sense of safety.

A plain precision/recall benchmark never reveals this. It tells you the caller's overall accuracy. It does not tell you whether you can trust the per-call confidence the caller hands you, which is the thing your filters actually consume.

A worked illustration

I built a small framework that computes exactly this: variant-calling-calibration-benchmark. It evaluates a caller on four layers: concordance (precision/recall/F1 vs GIAB), stratification (the same, split by genomic difficulty), calibration (the curve and ECE above), and a filtering-as-abstention analysis (more on that below).

To show the method end to end without a multi-gigabyte download, the repository ships a synthetic caller, built deliberately with the kind of miscalibration real callers exhibit: overconfidence concentrated in difficult regions and indels. I want to be plain that these demo numbers are synthetic; the point is to illustrate what the framework measures, not to report a discovery. Running it on a real caller is one command, shown at the end.

On that synthetic caller, the calibration layer reports:

Expected Calibration Error (ECE): 0.14
Mean stated confidence:           0.95
Empirical precision:              0.81
Verdict:                          OVERCONFIDENT

The caller asserts 95% mean confidence and delivers 81% precision. The calibration curve shows the points sitting below the diagonal across the whole confidence range. That gap is invisible to an F1 score and obvious the moment you plot stated confidence against empirical precision.

The stratification layer shows where the gap lives: near-perfect concordance (F1 ~0.99) in high-confidence regions, collapsing to ~0.77 in segmental duplications and low-mappability regions. The overconfidence is worst exactly where the genome is hardest.

Filtering as an abstention decision

Here is the part I find most useful, and it comes from thinking about callers the way you would think about any model that can decline to answer.

A QUAL filter is not just a quality cutoff. It is a deferral decision: every call below the threshold is one the caller is choosing not to commit to. Raise the threshold and you remove false positives (good) but also discard true positives (costly). The framework sweeps thresholds and finds two reference points:

the threshold that maximises retained F1 (best overall), and
the lowest threshold that reaches a target precision such as 99%, the point where what you keep is trustworthy enough to act on without review.

The distance between those two is a direct measure of the safe-versus-decisive tradeoff. On the synthetic caller, reaching 99% precision requires filtering all the way up to QUAL ~90, which discards most calls. The caller can be made trustworthy, but only by being made nearly silent. That is a property worth knowing before you deploy it, and a plain benchmark will never tell you.

Validating the evaluator itself

A benchmark whose metric is wrong is worse than no benchmark, so the ECE implementation is tested against inputs with known calibration. A synthetically honest caller (true-positive probability set exactly to the QUAL-implied confidence) must yield ECE near zero. A synthetically overconfident caller (high QUAL, 50% actually true) must yield large ECE. Both are asserted in the test suite. The metric is proven correct before it is trusted to judge anything.

def test_well_calibrated_low_ece():
    report = compute_calibration(well_calibrated_set())
    assert report.ece < 0.05

def test_overconfident_high_ece():
    report = compute_calibration(overconfident_set())
    assert report.ece > 0.2

Running it on real data, the right way

The framework's built-in matcher uses exact position-and-allele comparison, which is fine for normalised VCFs but stricter than it should be on real data, where variant representation differences are common. For real GIAB benchmarking the correct tool is hap.py, the GA4GH/GIAB field standard, which does proper normalisation and haplotype-aware matching.

So the real-data path delegates matching to hap.py and runs the calibration and abstention layers on its annotated output:

# hap.py does the matching (the hard, solved problem)
hap.py truth.vcf.gz your_caller.vcf.gz -f truth.bed -r GRCh38.fasta -o happy_out

# this framework adds the calibration and abstention analysis hap.py lacks
python3 -m src.run_benchmark --happy-vcf happy_out.vcf.gz --caller-name gatk-hc

That division of labour is the point: hap.py answers "how accurate, where," and this framework answers "is the confidence honest, and where should the caller stop trusting itself." The second question is the one almost no variant-calling benchmark asks.

The takeaway

Accuracy and calibration are different properties, and your filters depend on the second one. A caller can have a respectable F1 and still lie to you about its confidence in exactly the regions where being lied to costs the most. Measuring calibration is cheap, it borrows a standard idea from ML evaluation, and it tells you something a precision/recall table cannot.

I applied this same way of thinking, measure honesty of confidence, not just accuracy, to large language models interpreting clinical variants in a companion project. Different model, same question. It turns out to be the more interesting question in both cases.

Code, tests, and the full four-layer framework: github.com/gbadedata/variant-calling-calibration-benchmark

If you benchmark variant callers and have never plotted stated confidence against empirical precision, try it on your own GATK or DeepVariant output. I would be curious whether your caller is as honest as you assume.

References

Krusche P, et al. (2019). Best practices for benchmarking germline small-variant calls in human genomes. Nature Biotechnology 37(5):555-560. doi:10.1038/s41587-019-0054-x
Zook JM, et al. (2019). An open resource for accurately benchmarking small variant and reference calls. Nature Biotechnology 37(5):561-566. doi:10.1038/s41587-019-0074-6
Wagner J, et al. (2022). Benchmarking challenging small variants with linked and long reads. Cell Genomics 2(5):100128. doi:10.1016/j.xgen.2022.100128