Originally published at han-co.com · Part of the "Basics" strand of my Credit & Finance Data Science series. (The original has hand-drawn diagrams.)
Across nine pieces up to Part 8, we've covered why credit data science is different: selection bias and time, asymmetric costs, causation, validation and governance, interpretability and fairness. All of it was concepts.
But concepts don't run by themselves. A model only lives on top of a pipeline. Without the plumbing that gathers the data, builds the features, trains, and ships the scores, even a very good model stays an idea on paper.
This piece is about that plumbing. It isn't glamorous, yet a large share of the actual work goes here. And in this field, consistency and reproducibility come before performance.
The tool you hold longest is SQL
Coming in from a modeling background, it's easy to assume you'll mostly write Python. In practice you spend longer in SQL.
That's because most features are built there. Average spend over the last three months, delinquency counts over six months, the trend and volatility of a balance — these are all time-window aggregations. Pulling them out of a transaction table with tens of millions of rows happens in a large warehouse like BigQuery, and window functions become your main instrument.
Writing good queries here doesn't just mean getting the right answer. How you set up partitioning and clustering can change the cost of a query that returns the same result by a factor of tens. Adding one feature can become a monthly-bill question.
Leakage usually creeps in through the join
What really matters in SQL, though, isn't performance. It's that query design is the front line against data leakage.
In Part 0 I said time flows one way, and in Part 3 that this is why we use out-of-time validation. When that principle comes down into code, it looks like this: every join and aggregation must be done as of the decision point. This is called a point-in-time join.
It sounds obvious and it breaks constantly. The most common accident is attaching a current-state table as-is. You pull the credit limit from the customer master, but it's the limit as of today rather than as of the application. Or you attach delinquency history and sweep in delinquencies that happened after the application.
What makes this accident dangerous is that validation won't catch it. A leaked feature fits just as well in training and in validation. Splitting out-of-time doesn't help if both sides are contaminated. So when the numbers come back unusually good, the right move is to re-read the join before celebrating. In the field, an AUC above 0.95 is usually not a reason to cheer but a reason to be suspicious.
Python comes after that
Once the data is a table, you're in Python.
You process with pandas, and move to polars when the data gets large. Models are mostly scikit-learn, LightGBM and XGBoost, with statsmodels when you need statistical explanation.
Credit has its own domain-specific tooling. The WOE transforms, binning and scorecard scaling from Part 4 are faster and safer with a library like optbinning or scorecardpy than written from scratch each time. For the causal inference of Part 6 there's DoWhy, EconML and CausalML; for the interpretability of Part 8, SHAP is effectively the standard.
The list of tools changes quickly. What matters is knowing which kind of tool a given problem needs.
When training and serving drift apart
Once the model is built, you have to actually ship the scores. And this is where a quiet failure often happens.
In training you built features with batch SQL, but in serving a real-time API computes the same-named feature slightly differently. Did "last three months" mean 90 days or the last three calendar months? Were missing values filled with zero or with the mean? The name is the same; the value isn't.
This is train-serve skew. Validation looks fine, production scores come out strange, and tracking down the cause can take weeks. In Part 7 I said you monitor with PSI after deployment; often enough, a PSI spike turns out to be this skew rather than a change in customers.
The fix is to keep the definition in one place. A feature store registers each feature's computation rule once, and both training and serving fetch the same definition. Even without bringing in a heavyweight system, simply collecting feature definitions in one piece of code that both sides call will prevent much of this.
Reproducibility is a precondition for validation
In Part 7 I said a model you can't reproduce can't be validated or audited. In practice, that reproducibility is built out of these tools.
Code goes in Git, of course, and library versions are pinned so the environment can be rebuilt. Which experiment ran with which settings and produced which numbers goes into an experiment tracker like MLflow. And what matters especially in finance is versioning the data and the model. You have to be able to trace, months later, exactly what data the model approved half a year ago was trained on.
If a model registry records which version was approved when and went into production, the audit trail from Part 7 follows naturally. The governance documentation and the engineering end up pointing at the same thing.
Data quality is model quality
Last is data quality.
Missing values, outliers and duplicates are the basics. What's scarier in practice is a definition changing quietly. An upstream team changes a code scheme and tells no one, and from some day forward the distribution of one feature is wholly different. The model doesn't know, and keeps scoring.
So you put a data contract in place that states the schema and the expected value ranges, and you check freshness — whether the data arrived on time. Credit adds one more. Information from credit bureaus has a reporting lag, which means a delinquency that happened yesterday may not be in today's data. Build features without accounting for that lag and, once again, what you saw at training time differs from what you see at the actual decision.
So: consistency before performance
This piece was closer to plumbing than to concepts. But this plumbing holds up the previous nine. Break point-in-time correctness and the validation of Part 3 loses its meaning; let training and serving drift and the monitoring of Part 7 points at the wrong thing; lose reproducibility and the independent validation of Part 7 becomes impossible.
That's why, in this field, confirming that a join is point-in-time and consolidating feature definitions usually buys you more than squeezing another 0.01 out of the model.
One piece left. However well you build it and however well you run it, a model does nothing until it becomes someone's decision. In the final piece I'll close with turning a score into a decision, and conveying that result to other people.
I write han-co.com, a blog on credit and finance data science, in the language of practice. New posts by email: https://han-co.com/en/blog/
Top comments (0)