Originally published at rantum.xyz, which is the canonical source.
Almost any predictive model can be made to look good on historical data. The hard part is a backtest that reports what was actually knowable at decision time, so the number survives contact with production. Here is the discipline we hold every model to at Rantum, and the worked example where it walked a flattering 5.3x result down to an honest 2x.
The challenge
Every predictive product is sold on a backtest: "our model would have caught X% of these events in advance." The trouble is that a backtest is the easiest thing in machine learning to fake, usually by accident. Read a feature that was only knowable after the fact, shuffle your train and test sets at random, or report accuracy on an event that almost never happens, and you can produce an impressive number that means nothing. The model then ships, and the edge that looked real on paper quietly disappears in production.
The gap between a flattering backtest and a defensible one is not a modeling trick. It is a set of disciplines about time: what was knowable, when, and whether the test respects that. Below is the checklist we run on every predictive engagement, and a worked example, a teardown-probability model inside AddressIntel, our real-estate intelligence platform, where following it changed the answer.
What we built
A repeatable point-in-time backtesting discipline, applied here to predicting which aging houses on the San Francisco Peninsula get demolished and rebuilt. Teardowns are rare, well under 1% of parcels a year. The outcome is only visible in fragmented municipal permit records, and the features tempt you to cheat at every turn, which makes it a good stress test for a method meant to travel to any domain with rare events and messy history. Five rules do the work.
1. Score as-of T: features from data dated at or before T, nothing else
Pick a set of historical scoring dates T, and at each one, build every feature using only information that existed on or before that date. This sounds obvious and is constantly violated, because the most predictive-looking features are exactly the ones that leak. A teardown changes the structure: score a parcel as of 2018 but read its year_built, square footage, or listing photos as they stand today, and every one of those attributes describes the new house that replaced the old one. The feature would silently encode the answer. Only dated history, assessment rolls, sales, and permits, each stamped with the year it was true, reconstructs cleanly as-of T. Enforcing this disqualifies most of the obvious features before a single metric is computed.
2. Time-based folds, never random cross-validation
Standard k-fold cross-validation shuffles rows at random into train and test sets. For anything with a time dimension, that is leakage by construction: the model trains on rows from 2020 and is tested on rows from 2018, learning from a future it would not have had. Pool observations across several as-of dates instead, and split on time: fit on T at or before 2018, test on T = 2020. Use enough distinct dates, we like six or more, that no single lucky window can carry the result, and report the metrics per fold so temporal instability shows up instead of hiding in an average.
3. Lead with the base rate; measure lift, not accuracy
When positives are rare, accuracy is worse than useless: a model that predicts "no teardown, ever" is 99%+ accurate and completely worthless. The honest frame is the base rate first, then lift (how much richer the top-ranked slice is than chance), alongside precision-at-k, capture (recall) at k, and the area under the precision-recall curve. Because positives are few, bootstrap a confidence interval on every figure so you don't over-read noise.
# Rare positives (teardowns run <1% of parcels): accuracy is a trap —
# "never a teardown" scores 99%+. Lead with base rate, then lift.
def evaluate(observations, score): # observations: (features, label), each built as-of T
base = mean(label for _, label in observations) # e.g. 0.0072
ranked = sorted(observations, key=lambda o: score(o[0]), reverse=True)
k = len(ranked) // 10 # top decile
precision_at_k = mean(label for _, label in ranked[:k])
return base, precision_at_k / base # lift is the headline
# Pool across as-of dates T with TIME-based folds — fit on T <= 2018,
# test on T = 2020. Random k-fold shuffles future rows into training: leakage.
4. The label must be independent of the feature it validates
This is the rule that saved the model from shipping a lie. The first teardown label was a cheap proxy: California's Proposition 13 caps assessed-value growth except on new construction, so a teardown-rebuild shows up as a sharp jump in a parcel's building value. Scored against that label, a single heuristic (sort parcels by their land-to-improvement value ratio) looked spectacular: a 5.3x lift at the top decile, capturing over half of all "teardowns," on 17,589 observations at a 0.72% base rate. It read as an undefeated baseline.
It was an artifact. The proxy label is a building-value spike, and the winning feature is land divided by building value. Both are functions of the same number, so scoring by the ratio was, in effect, scoring against a shadow of the label it was supposed to predict. We caught it two ways. First, a direct cross-validation: of 140 proxy-flagged rebuilds, exactly one had a confirming demolition or new-build permit at the same parcel. Second, rebuilding the label from real, independent data (actual San Jose demolition and new-single-family permits, keyed to the parcel number across a 36,896-parcel universe). On that clean, non-circular test the ratio's edge as a standalone score evaporated to no signal at all, though the underlying lot-size economics still carried a smaller, defensible signal (about 2x lift), which is the number the shipped model stands on.
5.3x lift on the circular proxy label, then 0x for the same feature against an independent label.
The rule generalizes past this one case: whenever a label is derived from the same source as a feature (a churn label inferred from the activity you also feed the model, a fraud label built from the rules you're scoring against), the backtest measures arithmetic, not prediction. Build the label from a source the features can't see, or the number is a mirror.
5. Pre-register the go/no-go, and build a gate that lets you fail cheap
Write down the bar for success before you look at any results, so you can't move the goalposts to fit the number you got. Ours here was concrete: on out-of-sample dates, a top-decile lift of at least 5x, and a material margin over the best single heuristic (not within noise), and a positive median lead time so the call is early enough to act on. Just as important is a cheap feasibility gate up front. Before running any metrics, we counted teardown positives in the candidate windows; the pre-set rule was that fewer than roughly 30 to 50 positives is dead on arrival. The first pass returned zero strict positives against a ceiling of 19, so the metrics were never run on noise, and the real work, assembling an independent permit label, was correctly identified as the project rather than a detour.
Backtest is not the same as the live model
The final piece of the discipline is knowing what the backtest does and does not license. The leakage that disqualifies year_built, square footage, and photos is a property of scoring a parcel that was already rebuilt in the historical window. Score a house that is still standing today, and those same attributes describe the actual candidate structure: no leak, and they are strong features. So the backtest validates the economic and assessment layer under strict point-in-time rules, while the live product can honestly be richer than the backtest could ever prove. Conflating the two is how teams either cripple a live model with backtest-only caution or ship a backtest number the production features quietly violate.
What it shows
The unglamorous core of trustworthy predictive ML: reconstructing features as-of a decision date, splitting on time rather than at random, measuring rare events by lift instead of accuracy, keeping the label independent of what it validates, and pre-committing to a bar. None of it is exotic. All of it is routinely skipped, because skipping it produces better-looking numbers. It is the same through-line as the rest of Rantum's work (finding real signal in messy data) turned inward on the measurement itself.
What it proves to a client
That we will tell you whether a model works before you bet a roadmap on it. This case study is the proof: a 5.3x result that looked ready to ship as a "proprietary teardown score" got walked back to a defensible 2x lot-size signal once a clean test was possible, and we would rather deliver the smaller true number than the larger false one. Anyone can produce a flattering backtest by leaking the future into the features; the value is in the rigor that refuses to. For a business sitting on fragmented, underused data, that is the difference between a predictive product you can defend to a customer and a demo that falls apart in production. The method transfers directly to any domain with rare outcomes and messy history: insurance, lending, fraud, marketplaces, logistics, churn.
Andrew Maury is the founder of Rantum, a senior data science and ML studio that turns messy, fragmented, and adversarial data into models, APIs, and products that ship. Read the full case study at rantum.xyz.
Top comments (0)