Every model in this series so far answered the same question: what value of y goes with this x. Day 61's ARIMA forecast the next value. Day 60's survival model forecast a time. Day 1's regression forecast a number.
Not one of them can tell you what would happen if you intervened — because none of them was ever asked.
Today's question is different in kind, not degree: what would this outcome have been under the other treatment?
That quantity — the potential outcome Y(1) against Y(0) — is not a prediction problem. For every individual, exactly one of the two is observed and the other is permanently missing. That is the fundamental problem of causal inference, and it is not a data-collection failure you can fix with more rows.
So "correlation is not causation" stops being a slogan and becomes an estimand problem. The thing you can compute, E[Y|T=1] - E[Y|T=0], and the thing you want, ATE = E[Y(1) - Y(0)], are two different numbers.
Live, with a scoreboard: https://dev48.infy.uk/ml/day62-causal-inference.html
There is a scoreboard, because the data know the answer
Every unit on the page is generated with both potential outcomes from the same noise draw. So the true average treatment effect is known to machine precision, and every estimator gets marked against it.
A confounder X drives both who gets treated and what happens to them. The naive difference in means is then off by exactly:
E[Y(0)|T=1] - E[Y(0)|T=0]
That is not a vague "bias" — it is a specific quantity, and the page prints it next to the error you actually observe. They match.
Six ways to rebuild the comparison
Randomisation fixes confounding by cutting one arrow. When you cannot randomise, the propensity score e(x) = P(T=1|x) gives you four more routes, all implemented from scratch:
- IPW — reweight the sample into the trial you could not run.
-
Matching — pair treated with control at similar
e(x). -
Stratification — block on
e(x)and pool within-block effects. - g-computation — model the outcome, then predict a world that never happened.
- AIPW / doubly robust — combine a propensity model and an outcome model, and you only need one of the two to be correctly specified.
The propensity model is a real logistic regression fitted by IRLS, not a library call.
Balance is the diagnostic that matters
A propensity model with excellent AUC can still be useless. What you actually need is balance: after weighting or matching, do the covariate distributions of the two arms line up?
The page reports standardised mean differences before and after, because that — not classification accuracy — is the thing that tells you whether the comparison you rebuilt is a fair one.
What matching quietly does to your estimand
Discarding unmatched units does not just cost you sample size. It changes the question you are answering — you are no longer estimating the effect for the whole population, but for the region where the arms overlap. The page shows the discards and names the estimand that survives them.
The assumption you can never test, priced
Unconfoundedness is not verifiable from the data. Ever. So instead of asserting it, the page runs a sensitivity analysis: how strong would an unmeasured confounder have to be to overturn the conclusion?
That converts an untestable assumption into a number a reviewer can argue with — which is the most honest thing you can do with it.
All six estimators, one score, and an explicit account of what each one costs. Written from scratch, running live.
Top comments (0)