DEV Community: Han-co

You must be able to say why it decided: interpretability and fairness in credit

Han-co — Tue, 04 Aug 2026 14:19:43 +0000

Originally published at han-co.com · Part of the "Basics" strand of my Credit & Finance Data Science series. (The original has hand-drawn diagrams; the text below is identical.)

Up through Part 7, we've talked about building a good model (Part 4), evaluating it honestly (Part 5), setting policy with causation (Part 6), and validating that model and keeping it alive (Part 7). Now the last axis of trust remains: you have to be able to explain why it decided the way it did, and to answer whether that decision is fair.

In Part 0, among the reasons this field differs from ordinary ML, I named "interpretability isn't a choice, it's a duty" and "regulatory and governance overhead is always underneath" (points 6 and 7). This piece takes those two head-on. In other fields, being able to answer "why?" is a nice bonus; in credit, not having it is often illegal, or stops the release outright.

Explanation isn't a bonus, it's a duty

First, why interpretability is a duty. In credit, a model turns down someone's loan, trims their limit, raises their rate. A model that makes these decisions has to be able to explain why it judged the way it did. But whom you owe the explanation to differs by country.

At one end is the US. There's a requirement to notify a rejected applicant of the main reasons for denial, so the duty to explain points at the consumer. At the other end is Japan. A card company's underwriting criteria are private, and there's no obligation to tell a rejected applicant why. About all an applicant can do is request disclosure of their own credit information from a bureau like CIC and check that.

But not notifying the consumer doesn't mean no explanation is needed. The direction is just different. In Japan too, a model has to be explainable to the regulator; where the model becomes the legal basis itself, as with the payment-capacity estimate (支払可能見込額) under the Installment Sales Act, you have to be able to defend that judgment; and internal validation and audit (Part 7) also ask "what's the basis for this score?" So in credit a black box isn't a cool piece of cutting-edge tech, it's a risk in itself. However good the performance, if you can't say why it came out that way, you can't use it.

Transparent models and post-hoc explanations

There are broadly two ways to make a model explainable.

One is to use a transparent model from the start. Scorecards, logistic regression, shallow decision rules — their structure is the explanation. The WOE and scorecards from Parts 4 and 5 are the classic case: which feature moves the score up or down, and by how many points, is right there in a table. Models like this are often called glass boxes.

The other is to use a complex model for performance but attach a post-hoc explanation. The standard is SHAP. Grounded in game theory, it decomposes, in a consistent way, how much each feature contributed to this prediction. There are similar tools like LIME, partial dependence, and permutation importance.

It helps to distinguish two kinds of explanation here. Global interpretation looks at what the model as a whole weighs; local interpretation looks at why this particular person got this score. In credit, local interpretation matters especially. To send an adverse-action notice, you ultimately have to be able to say, one individual at a time, "you were declined for these reasons."

One honest caveat: a post-hoc explanation doesn't really open up the black box, it approximates it from the outside. SHAP values aren't the perfect truth either. So where possible you favor an interpretable structure from the start, and even when you use a complex model, you don't bolt the explanation on in a rush afterward — you design it in from the beginning.

A device often layered on top of this is the monotonicity constraint. It forces the model to keep to a direction that fits common sense and regulation, such as "a higher income must not lower the score" (in Part 0 I mentioned monotonicity as a condition for stability). Giving up a little performance to keep a sensible direction is often the better trade for explanation and trust.

Not using them doesn't make you fair

Now fairness. The starting point is clear. You must not discriminate on protected attributes like sex, race, or religion. This is prohibited by law, and putting those variables into the model directly is plain discrimination.

The trap is that dropping those variables doesn't make you fair. If other variables like ZIP code, occupation, or spending pattern are correlated with a protected attribute, the model learns the same discrimination by a detour, through that proxy. For example, in a society where ZIP code is strongly tied to race, using ZIP code alone without race still ends up as racial discrimination. This is called proxy discrimination.

At its root it connects back to Part 0. The data we hold already has the past approval policy and society's biases soaked into it. The model learns that data, so if you train it thoughtlessly, it inherits and reproduces the past discrimination as is. So "just drop the protected variable and it's fair" is naive; you have to look at the proxies too.

Fairness has no single definition

So how do you measure whether it's fair? The problem is that fairness has more than one definition.

Just to name a few: there's a definition that says equalize the approval rate across groups (demographic parity); one that says equalize the approval rate among the people who actually repay (equal opportunity); and one that says equalize the calibration of predicted probabilities across groups (calibration within groups — the calibration from Part 5). All plausible, yet they say different things.

Take the first, the instinct to compare approval rates across groups. There's a rule of thumb long used in the US: the four-fifths rule, often called the 80% rule. If one group's approval rate falls short of 80% of the highest group's rate, that's treated as a sign of adverse impact. It originally came out of hiring and selection, but it's frequently referenced in lending-discrimination judgments too. That said, it isn't a yardstick that settles discrimination; it's closer to a first sign that you should look harder. You can clear 80% and still be discriminating in substance, and fall short of it yet be explainable by a legitimate reason.

And there's an uncomfortable fact known mathematically. When the base rate — the actual default rate — differs across groups, you can't satisfy all these definitions at once. Line one up and another goes off. So "what counts as fair" is a value judgment technology can't make for you; it's something people have to choose to fit the context and the jurisdiction.

It can conflict with accuracy, too. Reaching for fairness often means giving up some performance. Mitigation can act on the data before training, on the objective during training, or on the output after training, but none of it is free.

So here too the honest stance is the same as in Part 6: state which fairness criterion you chose and why, and be aware of what that choice gives up. And this area must be handled together with legal and compliance. Being technically fair doesn't mean it's legally lawful.

Regulation holds all of this up

The explanation requirements and fairness demands so far aren't just well-meaning advice. A good deal of it is enforced by law and regulation.

The form differs by region. The US, through fair-lending law, requires adverse-action notices and prohibits discrimination; Europe, through its data-protection rules, addresses explanation of automated decisions, and its new AI regulation classifies credit scoring as high-risk. In Japan, the duty to compute a payment-capacity estimate (支払可能見込額) under the Installment Sales Act attaches to card issuance and limits, so the model becomes a legal basis directly (I flagged this in Part 0), and the Personal Information Protection Act governs data use. In Korea, the Credit Information Act and the Personal Information Protection Act cover explanation of, and objection to, automated decisions.

Privacy overlaps here. Which variables you put into the model is, directly, the question of how you use personal data. Purpose limitation, consent, data minimization, retention periods, access control — all of it is tangled up with feature selection. So choosing a single feature becomes a data-governance decision.

In the end, the "you can't deploy a model freely" from Part 0 meets the governance of Part 7 right here. Regulation and governance are always laid underneath all of this.

So, it's trust

With that, we've reached the last piece in the Basics series of why credit data science is different. Predicting well is the entry ticket (Part 5); handling causality is skill (Part 6); honestly validating the model you built and keeping it alive for the long haul is one axis of trust (Part 7); and explaining why it decided the way it did, and taking responsibility for whether that decision is fair, is the other axis of trust.

In Part 0, to the question of whether generative AI would just do all of this for us, I answered that you need it more, not less. Because in the face of requirements to be explainable, to be fair, and to pass regulation, a black box is structurally blocked. So the person who knows why explanation is required, what fairness is, and how you validate stays in the seat that judges what the automation produced.

That's the end of the conceptual part. From the next piece, we move to the side that actually makes these concepts run. A model only lives on top of a pipeline, and reproducibility and consistency come before performance — on to data engineering and tooling.

I write han-co.com, a blog on credit and finance data science, in the language of practice. New posts by email: https://han-co.com/en/blog/

Deployment isn't the finish line: validating and monitoring credit models

Han-co — Fri, 10 Jul 2026 08:48:54 +0000

Originally published at han-co.com · Part of the "Basics" strand of my Credit & Finance Data Science series. (The original has hand-drawn diagrams; the text below is identical.)

Up through Part 6, we've talked about building a good model and then designing policy with it. In finance, though, this is where the real work begins.

In Part 0 I mentioned in passing that "a model starts aging the moment it ships," and that "there's a reason things are slow." This piece is where I unpack that reason. In other fields, deploying a model feels like crossing the finish line; in credit, deployment is closer to the starting line. If building it took a few weeks, validating, overseeing, and watching over it afterward takes months — sometimes years.

Here's what this piece covers: having someone else validate the model independently before deployment, deploying itself carefully, and — after deployment — keeping watch and retraining it once it ages. And, wrapping all of it, documentation and governance.

The builder shouldn't be the validator

First, a concept: model risk. It's the umbrella term for the losses, reputational damage, and regulatory exposure that arise when a model is wrong or misused. A single model flows all the way through to provisioning and capital (Part 0), so when it's wrong, the whole firm wobbles. That's why finance treats a model as an asset that carries risk, and the framework for managing it is called model risk management (MRM).

There's one principle of MRM worth remembering: the person who builds the model and the person who validates it must be different people. This is usually called the three lines of defense. The first line is the team that develops and runs the model; the second is validation and risk management, independent of the first; the third is internal audit. "I validated my own model and it came out great" may fly in other fields, but not in finance. The builder has a hard time seeing the weaknesses of their own model. So validation is always done by a different set of eyes.

What validation looks at

When an independent validator receives a model, they look at roughly five things.

First, conceptual soundness. Do the methodology, the assumptions, and the variables make sense in the first place? Even if performance looks good, a relationship that defies common sense — "higher income means higher default risk" — is a coincidence in the data, not a signal you can trust.

Second, quantitative validation. Confirm discrimination, calibration, and stability in numbers — the three axes from Part 5. Is the AUC good but the PD wrong? Does it stay stable as time passes?

Third, sensitivity and stress testing. Does the score stay put when you nudge the inputs a little, and how does the model behave under a bad scenario like a recession? A model that only fits well in good times collapses exactly when you need it.

Fourth, benchmarking. Does this complicated model actually improve on a simpler one, or on the model already in place? If it's complex and doesn't improve much, that complexity is just risk.

Fifth, reproducibility. Run it again on the same code and data — do you get the same result? A model you can't reproduce can't be validated or audited.

You don't switch it on all at once: shadow mode

Passing validation doesn't earn a model an immediate spot in production, either. The standard safeguard when introducing a new credit model is shadow mode.

Shadow mode means running the new model alongside the live one, computing its scores without letting them drive any real decision. You approve and reject with the existing method, while recording — for months — what the new model would have decided. Only after you've confirmed the new model runs stably and doesn't diverge wildly from the existing results do you actually switch over.

In a similar spirit, you might compare a champion and a challenger in parallel, or do a gradual rollout that turns the model on for a subset of customers first. Either way, there's one thing in common: a rollback plan laid down in advance so you can undo it the instant something goes wrong. The principle isn't "switch on the high-performing model fast" — it's "switch on safely, even if it's wrong."

Models age: monitoring and retraining

After deployment it isn't over — the longest stretch is only just beginning. It's the aging I mentioned in Part 0. While a model trained on 2024 data scores 2026 customers, the economy, rates, and customer behavior all change. The distribution shifts, and the model slowly ages.

So after deployment you keep watching from several angles. First, how far the inputs have drifted from training time, with PSI and CSI — the stability metrics from Part 5. You track performance too, of course. But there's a difficulty peculiar to credit here. Since a default label is only settled 12 to 24 months later (Part 0), you can't see performance decay in real time. So you run backtests — matching predictions against actuals — belatedly, as labels accumulate, and in the meantime you catch warning signs early with leading indicators like PSI. You also watch whether the predicted default rate matches the actual, whether one particular segment is drifting badly, and whether the data pipeline has quietly broken.

And once you judge the model has aged, you retrain it. When to retrain — the trigger — is decided in advance too: when the distribution has moved a lot, when performance has dropped, when the policy changed, when a new product launched. The point is not to retrain on a whim, but to define the cadence and the procedure ahead of time and follow that governance. Because changing a model is itself a risk.

The documentation is the model

Finally, it's about leaving a written record of everything about the model. This part is surprisingly the crux.

Every model comes with development documentation: what it's for, what data and what methods and assumptions built it, what its limits are, and what the validation results and monitoring plan look like. To that you add an audit trail recording when what changed, plus version control and change management.

Here's the crux: documentation isn't cleanup you do after the model is finished — it's part of the development process. When the regulator asks "why this model?", when internal audit asks "what's the basis for this decision?", the documentation is the answer. A model you can't explain on paper is unusable in finance, no matter how well it performs.

So, there's a reason things are slow

In Part 0 I said the startup reflex of "ship it fast" doesn't travel well here, that there's a reason things are slow. This piece has been the explanation of that reason. Someone other than the builder validates; you switch on carefully; you watch it age and retrain; and you leave all of it in writing. It looks like a hassle, but it isn't a burden regulation forced on you — it's the infrastructure of trust. Because a single model judges one person's credit and can shake the firm's capital.

Predicting well is the entry ticket (Part 5); handling causality is skill (Part 6); and honestly validating the model you built and keeping it alive for the long haul — that is trust. Next time, I'll continue to the other axis of that trust: interpretability, the demand that you be able to explain why the model decided what it did, along with fairness and regulation.

I write han-co.com, a blog on credit and finance data science, in the language of practice. New posts by email: https://han-co.com/en/blog/

Where do rejected applicants go? Reject inference and rejectkit

Han-co — Tue, 07 Jul 2026 07:32:23 +0000

Originally published at han-co.com · Code: GitHub · Part of the "Deep Dive" strand of my Credit & Finance Data Science series. (The original also has a hand-drawn diagram.)

In Part 4 I touched on reject inference briefly. The point was that a model built on approved customers alone becomes biased once you apply it to the full applicant pool. This post is a record of writing that reject inference in code. I bundled the classic techniques together and, above all, built a library that also evaluates "whether this correction actually helps on my data." It's called rejectkit, and it's now on PyPI and GitHub.

You will never know how the rejected applicants turned out

Picture a loan review. An applicant arrives, you look at their information, and you either approve or reject. For the ones you approve, you learn the outcome a few months later: pay well and it's good, fall behind and it's a default. But the ones you reject were never given a loan, so there's nothing to repay. You will never know how they would have turned out.

That's where the trouble starts. To build a model that predicts whether next year's applicants will fall behind, you need labeled data, and the only labels you have belong to the people you approved. But approved customers are a skewed sample: they got through precisely because they looked fine in the first place. A model trained on them alone has a distribution that's out of step with the full set of applicants who actually show up at your door.

It's the same as a doctor who concludes "my treatment works great" from the patients who came back. The ones it didn't help never returned, so they never make it into the data. This is exactly the selection bias from Part 0 and the reject inference from Part 4.

And yet the Python tooling was empty

Reject inference has been a standard topic in credit risk for decades. And yet, when I went looking, the Python tooling was empty.

R has a package called scoringTools, but it isn't even on CRAN, only on GitHub. Python's scorecard libraries (scorecardpy, optbinning, toad) do the WOE/IV binning and logistic scorecards from Part 4 well, but they don't touch reject inference at all. What was left was one-off research code written for papers.

So I built rejectkit. There were two goals. One was to bundle eight classic techniques behind a single scikit-learn-style API. The other, the more important one, was to provide a benchmark that measures "whether this correction actually helps on my data." The truth is that "the value of reject inference is doubtful, and no technique is always superior" has been the field's long-standing conclusion. A paper literally titled "Reject inference — can it work?" was already out in 1993. So the real message of this library isn't "trust it and use it," it's "measure first."

Eight techniques, and the assumption that decides everything

The ways of bringing rejects back into training fall into roughly three families.

First, methods that manufacture labels for the rejects and fill them in (the augmentation family). Score the rejects with the approval model and cut at a threshold to assign labels (simple); split one person into a good version and a bad version, two rows with weights (fuzzy); multiply the default rate by score band by a weight to encode the practitioner's assumption that "a reject is worse than an approved customer at the same score" as a number (parcelling); or borrow the default rate of nearby approved customers with similar features (extrapolation).

Second, methods that correct without manufacturing labels. Train a selection model that separates approvals from rejects and give approved customers an inverse weight (IPW reweighting, the same logic as the bias correction from Part 2), or take econometrics' Heckman control function and fold it into the classification problem as an extra feature.

Third, semi-supervised learning. Leave the rejects as unlabeled data, attach pseudo-labels only to the high-confidence ones, and repeat the retraining loop (self-training).

Which technique works comes down, in the end, to a single assumption: what the rejection depends on.

If rejection depends only on observed features (MAR), a well-built model actually isn't all that biased.
If rejection also depends on the unobserved outcome (MNAR), say a past reviewer screened out bad applicants using information that isn't in the data, then a model that saw only approved customers is the most biased of all.

And here's an important limitation. The augmentation family leans on the biased approval model to guess the rejects' labels, so it can't pull itself out of a strong MNAR situation on its own. Which is why the right question isn't "which technique should I use" but "does reject inference even help in my situation right now," asked first.

How do you measure whether it helps

The fundamental difficulty of reject inference is this: rejects have no ground truth, so you can't grade the correction directly.

rejectkit sidesteps this. It takes data where you know every outcome, deliberately picks some rows to play the role of rejects, and hides (masks) their labels. Then it grades how well each technique recovers those hidden labels, scored on an untouched test set. The "Masked" in the name is this label hiding.

The key metric is auc_recovery. Zero means on par with the naive model that used approved customers only; 1 means recovery all the way to the oracle that used every label; negative means it made things worse.

You pick how rejections are generated from three options: mar, which depends only on features; mnar, which also depends on the hidden outcome (the harshest); and cutoff, which approves in order of lowest predicted risk (closest to a real credit policy).

So does it help: synthetic data vs. real data

I ran it on clean synthetic data with MNAR. The oracle is 0.820 and naive is 0.749, but the reject-inference techniques barely clear naive and several actually lose ground. fuzzy scrapes break-even at +0.001, parcelling degrades to -0.116, and only Heckman just about holds the naive line. When selection depends on the outcome (MNAR), it plays out exactly as theory predicts. Reject inference is no free lunch.

So let's check on real data. I applied it as-is to the Kaggle Home Credit dataset (about 300,000 rows, 8% default rate).

Model	AUC	auc_recovery
oracle (all labels)	0.741	1.00
naive (approved only)	0.568	0.00
parcelling	0.582	+0.084
extrapolation	0.582	+0.079
heckman	0.580	+0.071

Here it's the opposite. Ignore the rejects and you collapse from 0.74 to 0.57, and reject inference actually recovers 7–8% of that loss. What was nearly useless on synthetic data is worth having on real data. And even with the same dataset, switch to mar or cutoff and naive is already oracle-grade, so there's no room left to recover (auc_recovery comes back as NaN).

The conclusion lands in one place. Whether reject inference helps depends on the data, which is why you have to confirm it with a benchmark before you use it.

How to use it

Fitting reject inference onto a single model goes like this.

from sklearn.linear_model import LogisticRegression
from rejectkit import RejectInferenceClassifier

# X_accept, y_accept: approved customers and outcomes (1=default) / X_reject: rejects (features only)
clf = RejectInferenceClassifier(
    estimator=LogisticRegression(max_iter=1000),
    method="parcelling",
    method_params={"uplift": 1.3},   # assume rejects are ~30% worse than the same score band
)
clf.fit(X_accept, y_accept, X_reject)
pd_bad = clf.predict_proba(X_new)[:, 1]

Measuring whether it helps first is one line.

from rejectkit import MaskedRejectBenchmark

bench = MaskedRejectBenchmark(selection="mnar", accept_rate=0.6, random_state=0)
print(bench.compare(["fuzzy", "parcelling", "reweighting", "heckman"], X, y).round(4))

Inputs accept pandas, polars, and numpy alike. It also ships with diagnostics (per-feature PSI) and visualizations for seeing how far the approval and rejection distributions diverge.

Wrapping up

Train a credit model on approved-customer data alone and you get sample-selection bias. That's the problem from Part 0 and Part 4.
rejectkit bundles eight classic techniques for correcting it behind one API.
The real differentiator is the benchmark that measures whether the correction helps on your data. Reject inference isn't a cure-all and can even be harmful, so the core idea is to measure first rather than trust and use.
The same principle applies beyond credit, to any selection-bias setting where you only see outcomes for those who got through: hiring performance, or fraud detection where only the flagged cases get labeled.

And the most trustworthy answer to reject inference is, in fact, a different one entirely: an experiment that deliberately approves a small random slice to secure real outcomes. It hands you fact instead of an estimate. That thread continues into causal inference and experiments in Part 6.

Installing is one line.

pip install rejectkit

GitHub: github.com/HangilKim11/rejectkit
PyPI: pypi.org/project/rejectkit

Feedback, issues, and PRs welcome.

I write han-co.com, a blog on credit and finance data science, in the language of practice. New posts by email: https://han-co.com/en/blog/

Beyond prediction: causal inference and experiments in credit

Han-co — Tue, 07 Jul 2026 07:09:24 +0000

Originally published at han-co.com · Part of the "Basics" strand of my Credit & Finance Data Science series. (The original has hand-drawn diagrams; the text below is identical.)

So far this series has mostly been about getting "who is risky" right. Building good features, scoring with trees, checking whether those scores are well calibrated. But in practice the harder question, the one with real money riding on it, is a little different.

If we raise a limit, will defaults go up? If we chase a delinquent customer, will we recover more? If we cut the rate, will fewer people churn?

These aren't prediction questions, they're causal ones. In Part 0 I described the core of this field as "estimating the unobserved counterfactual," and this is exactly where that counterfactual walks in through the front door. And the moment you can answer these questions, a credit data scientist graduates from "the person who guesses who's risky" into "the person who designs policy."

Prediction and causation are different questions

Prediction asks: what is this customer's probability of default? Causation asks: if we raise this customer's limit, how much does that probability change? These are completely different questions, and the ways you answer them are different too.

The fundamental reason causation is hard is that you can never see two outcomes for the same person at once. What would have happened to this customer, whose limit we raised, if we hadn't raised it is something we will never observe. We only ever see half of the world. This is the general form of the story from Part 0, where I said we never learn how the customers we rejected would have turned out.

The catch is that in credit, the decisions we make are themselves interventions in the outcome. Whether we approve, how large a limit we grant, what rate we set all directly affect that customer's default. So the data in our hands isn't a world we sat back and observed, it's a world we already shaped by intervening in it.

Here's where a common trap appears. The data shows that customers with higher limits default less. So if we raise limits, will defaults fall? No. We simply handed the higher limits to customers who already looked creditworthy. The limit didn't lower defaults; a hidden cause, good creditworthiness, produced both the high limit and the low default at the same time. A hidden cause that moves two outcomes together like this is called a confounder. (I checked this directly with public data in the Deep Dive piece.)

Mistake a correlation for causation when you set policy, and no amount of data will keep you from getting the direction wrong. This is why in credit, causal inference isn't optional, it's mandatory.

The gold standard is an experiment

The way to turn a counterfactual from a guess into a fact is actually simple: randomize the intervention. Split the target customers at random into two groups, raise the limit for one and leave the other alone. Because the split is random, the two groups have the same creditworthiness and the same income on average. So whatever difference opens up later in default rate or profit is due to the limit and nothing else. This is a randomized controlled trial (RCT), and it's the gold standard of causal inference.

Credit has one special form of this experiment: deliberately approving, at random, a small share of applicants you would otherwise have rejected. In Part 0 and the rejectkit piece I said we can never learn how rejected customers actually turn out, and this is the one way to turn that counterfactual into a fact. Reject inference guesses at that blank space with assumptions; an experiment just lets you observe the answer. That's why I said the most trustworthy solution to reject inference is, in the end, a small randomized approval.

It isn't free, of course. Approving bad customers on purpose is a loss, and a randomized limit increase carries risk too. So you don't swing big, you run experiments small, with a controlled design. How large a sample you need to detect a difference statistically, whether to randomize at the customer level or the branch level, where the loss has to cross before you stop, and even the lag from default labels only settling 12 to 24 months later all have to be set before you begin.

When you can't experiment: quasi-experiments

You can't always run an experiment. Often it's too slow, too costly, or ethically or legally out of bounds. When that's the case, you approximate causation as best you can from observational data you've already accumulated. This is called a quasi-experiment.

First, propensity scores and inverse-probability weighting (IPW). This is a way of making the treated group and the untreated group resemble each other on their observed characteristics. It's the same logic as the bias correction from Part 2, or the reweighting in rejectkit. If you gave larger limits to customers with good creditworthiness, in effect you pair up and compare people of similar creditworthiness.

Second, regression discontinuity (RDD). This one fits credit especially beautifully. Underwriting has a cutoff score. If 630 is the line, an applicant at 631 and one at 629 are essentially the same person, yet only one gets approved. If the outcome jumps sharply across that boundary, that jump is the causal effect of the intervention that is approval. The underwriting cutoff becomes, in itself, a natural experiment.

Beyond these, the toolbox holds more: difference-in-differences (DiD), which compares before and after a policy change, and instrumental variables (IV), which use an outside factor that only affects the intervention as a lever. Quasi-experiments all lean, though, on an assumption of the form "there is no unobserved confounding." So the order is clear. Experiment when you can; when you can't, use a quasi-experiment but always state which assumptions you're standing on. And the honest move is to check, with a sensitivity analysis, whether the conclusion flips when those assumptions wobble a little.

Python has a few tools that help with this process. DoWhy lets you treat causation as a flow of "state your assumptions, estimate, and then refute by testing how much the result shifts if those assumptions are wrong." It essentially has you write your assumptions down in code and go knock on them yourself, which pairs well with the sensitivity check I just mentioned. EconML collects the latest methods for estimating treatment effects, especially the heterogeneous treatment effect (HTE) that varies from person to person (uplift, just below), such as double machine learning and Causal Forest. CausalML specializes in uplift modeling and targeting, handling everything from meta-learners to the Qini evaluation that comes later, in one place.

Beyond the average: uplift

So far we've been asking whether raising a limit increases profit on average. But the real question in practice goes one step further: for whom should we raise it?

Raising a limit for a customer whose profit won't grow either way just adds risk, which is pure waste. Conversely, raising it for a strong customer who couldn't spend for lack of headroom generates incremental profit. Estimating, at the individual level, this extra difference an intervention creates for a given person is uplift modeling. It's not the same as simply predicting who will respond. You're looking not for the person who would have acted anyway, intervention or not, but for the person the intervention actually changes.

The use in credit is clear. Raise the limit only for customers where the increase creates incremental profit, and intervene only on the delinquents where a chase actually lifts recovery. As for techniques, there are things like the T-learner, the X-learner, Causal Forest, and uplift trees.

One word of caution. You must not evaluate uplift with ordinary AUC. Since what we're trying to get right isn't whether someone defaults but the effect of the intervention, you have to look at dedicated metrics like the Qini curve or Uplift@k.

Here's the difference. AUC measures how well you rank whether this person will default, or respond. But what uplift wants is a ranking of how much the intervention changes this person, so what's being measured is different from the start. On top of that, since you can't see both the intervened and the un-intervened outcome for one person at once, there is no per-person ground truth at all, no answer key to grade one person at a time the way an ordinary metric does. So the Qini curve detours around this by working with groups. As you intervene on customers starting from the ones the model scored highest, it plots, cumulatively, how much faster the outcome gap between the treatment and control groups piles up than it would if you'd chosen at random. Uplift@k is the incremental gain piled up by the time you've reached the top k%. In a word, where AUC measures "how well do I guess who will default," Qini measures "when I intervened in the order I chose, how much more did I actually make than intervening on no one at all."

In Part 5 I said that when the objective differs the metric has to differ too, and this is where that principle applies most dramatically.

So, what expertise in this field really means

I'll say again that prediction is the price of entry and causation is the expertise. Every decision we make in underwriting is an act of intervening in the outcome, and only when you can estimate that effect honestly can you design policy.

To lay out the order: experiment when you can, even in a small way. When you can't, approximate with a quasi-experiment but state which assumptions you're standing on. Credit in particular has a natural experiment in the underwriting cutoff, so don't leave regression discontinuity going to waste. And beyond the average effect, use uplift to choose whom to intervene on.

This way of thinking is the root solution to reject inference, and the foundation for limit policy, collection strategy, and pricing. In the next part I'll carry this into governance and monitoring: how you validate, audit, and operate the models and policies you've built this way.

The record of debiasing the limit-causation question by hand on public data is in the Deep Dive, so if you'd like to see the concept confirmed hands-on, that's a good one to read alongside this.

I write han-co.com, a blog on credit and finance data science, in the language of practice. New posts by email: https://han-co.com/en/blog/

Can Google's TabFM beat GBM in credit? I tested it on public data

Han-co — Sun, 05 Jul 2026 13:33:15 +0000

Originally published at han-co.com · Code: GitHub · Part of the "Review" strand of my Credit & Finance Data Science series. (The original has hand-drawn diagrams; the numbers below are identical.)

This is the first piece in the "Review" strand of the series. It's where I take a newly released tool and, through a practitioner's eyes, measure whether it's actually useful for credit work.

Google recently released TabFM: feed it a table as-is and it predicts with no training and no tuning, and it claims that "zero-shot, it beats even a well-tuned GBM." If that really holds in credit risk — a place GBMs have dominated for years — it's a big deal. So I pitted it against a carefully built GBM on public loss data. To give the conclusion first: under the conditions I ran, a well-tuned GBM came out slightly ahead. But this was not a head-to-head under the same conditions the paper claimed its win, so I can't declare a winner — and more to the point, the key finding is that TabFM came right up against that best with no feature engineering and no tuning.

What TabFM is

TabFM is a foundation model for tabular data that Google released at the end of June 2026. The core idea is zero-shot. It doesn't retrain per dataset, tune hyperparameters, or engineer features. Instead it reads the training data in as a single context, and prediction is done in one forward pass. This is called in-context learning. Much as GPT takes a handful of examples in the prompt and answers, TabFM takes the whole table as a kind of prompt and answers on the spot.

What makes this possible is an enormous amount of pretraining done in advance. But unlike images or text, there isn't nearly as much public tabular data, so Google synthesized hundreds of millions of tables with structural causal models (SCMs) and pretrained on those. The learned architecture has three parts. It attends alternately over rows and columns to capture relationships between features, compresses each row into a dense vector, and then predicts on top of that via in-context learning. Think of it as an evolution of the TabPFN line, an earlier attempt at a tabular foundation model.

The selling point is clear: no feature engineering, no hyperparameter tuning. In other words, the effort you pour into a GBM in practice drops to zero. And Google reported that on the TabArena benchmark this zero-shot model outranks painstakingly tuned supervised models — GBM-family models in particular — on ELO. That benchmark gathers 38 classification and 13 regression datasets, ranging from 700 to 150,000 samples. The model is out now on Hugging Face and GitHub, and Google says it will land in BigQuery within a few weeks, callable from a single line of SQL (AI.PREDICT).

Why the experiment: can it be used on credit losses

I had one question. This model that supposedly beats GBM on general benchmarks — does it do the same on credit losses?

In an earlier piece I concluded that on tabular credit data it's tree-based boosting, not deep learning, that wins. TabFM claims to overturn exactly that conclusion, so it was worth re-checking. And credit isn't like a general benchmark. Default is a rare event; you need not just the ranking but the probability to be accurate; and you have to be able to explain the reasons behind a decision. Winning on general data doesn't guarantee winning on credit. So I checked directly, on public data, whether zero-shot TabFM beats a well-tuned GBM in credit too.

Data and models

The data is UCI's Taiwan credit-card default dataset: 30,000 people, 23 features, a 22% default rate. The main features are the last six months of delinquency status plus billed and paid amounts, and it's known in the literature as a signal-limited dataset where standard models tend to stall somewhere around AUC 0.77 to 0.78.

The heart of a fair comparison is separating the model from the features. Compare a "feature-engineered GBM" against "raw TabFM" directly and, even if you see a gap, you can't tell whether it's the model or the features. So I lined up several baselines side by side.

Name	What it is
GBM raw	LightGBM, no features, no tuning (the effort floor)
GBM tuned	LightGBM + engineered features + Optuna tuning
CatBoost / XGBoost	Strong baselines with features + tuning
TabFM zero-shot	Out of the box, single forward pass (the main event)
TabFM ensemble	Paper preset (derived features + calibration)

The GBM-family models got features like delinquency dynamics added and were tuned with Optuna. TabFM was left out of the box, with zero tuning. I trained every model at the natural rate so the probabilities would match the actual default rate. For metrics I looked, as credit demands, at both discrimination (ROC-AUC, PR-AUC, KS) and calibration (Brier, ECE). Since default is 22%, the imbalance isn't extreme, but I also checked how well the rare defaults are caught with PR-AUC. I validated with stratified 5-fold.

Results

Arm	ROC-AUC	PR-AUC	KS	ECE ↓	Time
GBM tuned (LightGBM)	0.789	0.566	0.443	0.010	548s
XGBoost	0.789	0.565	0.439	0.009	102s
CatBoost	0.788	0.566	0.444	0.011	1179s
TabFM zero-shot	0.785	0.558	0.441	0.022	503s
GBM raw	0.779	0.554	0.429	0.013	0.5s
TabFM ensemble	0.774	0.540	0.418	0.018	268s

There are three things to read off this.

First, a well-tuned GBM (0.789) edges out TabFM zero-shot (0.785). All three trees sit above TabFM, and the ordering is the same on PR-AUC too (TabFM 0.558, tuned GBM 0.566). The gap is 0.4pp, inside the fold standard deviation (0.006), so statistically it's close to a tie — but the direction is consistently GBM on top. "Zero-shot beats a tuned GBM" did not hold on this data.

Second, compare no-effort against no-effort and the story is different. TabFM zero-shot (0.785) clears raw GBM (0.779) — with no features and no tuning. As a fast baseline it's genuinely attractive.

Third, calibration is on par too. Trained at the natural rate, the trees also produce well-matched probabilities (ECE 0.010), and TabFM comes in slightly behind at 0.022. Neither is badly off.

And the three boosters converge between 0.7885 and 0.7891. That means the ceiling on this data — where changing the model and adding features and tuning still doesn't lift you — is around 0.79. Neither side got above it.

To sum up: at least in this experiment, TabFM didn't beat a well-built GBM, it came close to it with no effort. And you have to keep in mind that the conditions weren't the same.

Limits of this experiment

These results are only valid within a few conditions.

It's one dataset. A single Taiwan credit-card set can't guarantee generalization. On other loss data the ranking could change.
The signal is limited. With 23 features and a 0.79 ceiling, there wasn't much room for a model to pull ahead in the first place. On data with many features and rich signal it could come out differently.
One seed, and I couldn't do out-of-time validation. With no trustworthy time column I split by random stratification, but a real credit model is more rigorously validated by splitting on time.
TabFM ran on an 8GB GPU. Because of that I couldn't properly run the ensemble setup noted in the model limits below, so the TabFM numbers in the table above should be read as a lower bound.

Limits of the model itself

Leaving the experiment aside, here's what catches when you try to bring TabFM into practice.

It's a black box. Being a foundation model, there are no coefficients and no clear rules. For credit underwriting, where you have to disclose the reason for a decision and explain it to regulators, it's hard to use as-is. You can bolt on post-hoc explanations like SHAP, but the model itself doesn't become the explanation the way a scorecard does.
It needs a high-end GPU. The weights alone are 6.5GB, so without a GPU inference is more than ten times slower, and the paper preset (ensemble, large context) needs a 16GB-plus GPU to run properly. That's a sharp contrast with a GBM, which runs fine on a single CPU.
There's a ceiling on data and feature size. In-context learning reads the whole training set in like a prompt, but attention grows with the square of the context. So millions-of-rows credit data, or data with very many features, is hard to load as-is. That's exactly why this family of models was built for small- to mid-sized tables in the first place.
Inference is heavy. A saved GBM scores in one line, but TabFM re-reads the training data every time it predicts. In high-volume real-time scoring, that cost becomes a burden.
It's still new. As a just-released model, it has no track record in practice and no cases of regulatory acceptance.

What's next, and you too

TabFM is available on Hugging Face now, and within a few weeks it should be callable from a single line of SQL in BigQuery. The barrier to entry drops that much, so if you have a big GPU or the BigQuery integration opens up, it'd be worth running the ensemble preset properly and re-measuring on larger data. I'm especially curious how it does on small-sample data, feature-rich data, and time-based validation. This piece's conclusion is only the story of one Taiwan credit-card round, after all.

Here's my conclusion. In this one experiment GBM came out slightly ahead, but I didn't compete under the conditions the paper claimed its win (large, diverse data; an ensemble preset; ample GPU), so I can't declare that it "can't win." What is certain is this: it's a remarkably fast baseline that lands within 0.4pp of that best with no features and no tuning. For prototyping or a first baseline it's attractive right now, and for a production credit model that needs peak performance, accurate probabilities, and explanations, a well-tuned GBM is still the safe choice for now. It's too early to say the tree-based conclusion has fallen, but there's now a candidate that comes close with no effort.

Appendix: code and data

Data: UCI Default of Credit Card Clients (Taiwan), public data
TabFM source: Google Research blog · Hugging Face · GitHub
Code: github.com/HangilKim11/blog-research/tree/main/tabfm-credit
Reproducibility: stratified 5-fold

I write han-co.com, a blog on credit and finance data science, in the language of practice. New posts by email: https://han-co.com/en/blog/